Status for Andrew DeFaria

log_activity

Ray Tran wrote:

To summarize the three logging projects. The primary log directory is /projects/scm_tools_logs in San Jose. Please let me know if you have any input or suggestions

We need a global area that's truly global!

Turns out that /projects/scm_tools_logs doesn't exist. I assume you meant /projects/IT_SCM/logs. However /projects/IT_SCM doesn't exist on either ccase-rmna-3 nor ccase-sj1-1... So running a script that requires Clearcase such as this I cannot access the directory where I need to deposit the log files! Additionally I cannot access /projects/IT_SCM/SCM/lib for my libraries nor /projects/IT_SCM/SCM/cc where the script would be residing...

Activity logs

This is a log of who checked things out and into ClearCase and from what view. I think we can do this hourly and append to the log file so it doesn't take too long to run.

Turns out lshistory is pretty flexible and efficient. One does not need to do a cleartool find on all elements and then call lshistory on individual elements, etc. The lshistory command supports a -recursive parameter. But we really want to report things across vobs. The lshistory command also supports a -avobs parameter. Two limitations here... First lshistory needs to work the context of a view. Secondly, -avobs only works on vobs that are mounted.

I've coded up a script called log_checkins which will create a view (named default <- I can change this) if necessary and also mounts all vobs prior to calling lshistory. It also supports a -n parameter for number of days - thus -n 7 will produce a weeks worth of log files. Logfiles are named checkin.<date>.log where <date> is the date in YYYYMMDD format. So -n 1 (or not specifying -n) will produce a log file for the previous day (e.g. checkin.20051227.log) encompassing all activity on that day, while specifying -n 7 will produce 7 logfiles, one for each of the last 7 days properly dated.

The fields in the logfile (separated with ";") are:

1. <date>.<time> (<date> is YYYYMMDD and <time> is HHMMSS 24 hour format). This format makes the logfile easily sortable by date and time with sort(1).
2. <username> (Not sure where lshistory gets the username, normally is the same as the next field but sometimes it's different - e.g. "Clear Case Administrator")
3. <user>@<host> The user login name (e.g. adefaria) @ host (e.g. ltsjca-adefaria)
4. <event> The Clearcase event
5. <element> View extended path name

Clearcase Events

I ran log_checkins for the last 30 days and examined the log files. No filtering for only "check in" events were done. The range of Clearcase events are:

• checkout directory version
• checkout version
• create activity
• create baseline
• create branch
• create branch type
• create directory element
• create directory version
• create file element
• create hyperlink
• create label type
• create trigger type
• create version
• destroy activity in versioned object base
• destroy checkpoint "<checkpoint>" in versioned object base
• destroy element in versioned object base
• destroy hyperlink "<hyperlink> in versioned object base
• destroy sub-branch "<sub-branch>" of branch
• destroy type in versioned object base
• destroy version on branch
• import file element
• import hyperlink
• lock activity
• lock branch
• lock branch type
• lock trigger type
• lock versioned object base
• unlock activity
• unlock versioned object base

There are probably other event types. Also, in most cases the <element> field contains additional information that can be extracted. For example:

20051201.151257;drambo;drambo@PCRMNA-DRAMBO;create
version;/vobs/OnePhone/pub/Makefile@@/main/OnePhone_mainline_Intr/OnePhone_3.2_Intr_new/10

one can see that this was a creation of a version in the OnePhone vob of pub/Makefile. The part after the @@ is the branch information (/main/OnePhone_mainline_Intr/OnePhone_3.2_Intr_new) and version 10.

Although you asked for checked out things, checked in things are more interesting. In fact I'd say that all of the above Clearcase events are interesting and that we should just log all of it (and I'll change log_checkins -> log_activity and change checkin.<date>.log -> activity.<date>.log

Pulse check log

This is a script that should run a quick checkout and time the process. This should run at least every 15 minutes. I'm not sure if we need to set this up for each VOB or if we can just create a IT_SCM VOB and checkout from there. We then control how much data gets moved. For example in Perforce, I've created some file that are 100MB in size. I total the files and check this out. I now have control over the data.

Statistic logs

This is a script that would create two logs - one general data for each site, one for each VOB of each site.

The general data should have at the least - date_time;site;number of VOBs; number of total VOB size; number of snapshot views; number of dynamic views; number of file elements; number of directory element;...

The VOB specific data - date_time;site;VOB_name;size of database;size of source pool; size of devired object; size of cleartext; size of admin data

The directory structure that I envisioned is as such

/projects/scm_tools_logs
 
 - cc
      - 2005
          - 12
             - 01
 

I see little benefit in constructing such a directory structure, especially programmatically. Instead I place the YYYYMMDD in the file name of the log file in a format described above. This format is easily parseable, sortable and grepable and it also compresses nicely. A directory structure such as above could easily be created at a later time or sets of logfiles could be easily gzipped and tar'ed (e.g. mkdir 2005; tar -zcf 2005/activity.logs.gz activity.2005*; rm activity.2005*).

R/O Components & create_dev_snapview.pl

After speaking at length with Shivdutt about this and learning more about UCM, projects, streams and the like I see a problem here. Let me explain it based on how I see this...

With UCM you have projects which contain components which can be R/W or R/O. Projects also have streams, one integration stream (and an associated integration view) and one or more development streams (and their associated development views).

When a development view is created in a development stream it is populated with all of the components in the project based on baselines. The difference here is that you can use different baselines for the various R/W components but you cannot use different baselines for the R/O components. The baseline information for R/O components is kept "globally" at the project level and is not modifiable in the development streams. Since, theoretically, R/O components are not modifiable they considered shared for all development streams.

While R/O components can have their baselines changed (usually advanced) at the project level, the act of doing so affects all development streams (dynamic development streams will be affected immediately while snapshot development views at next update time).

So if we had create_dev_snapview.pl alter the baselines of the R/O components by retarding (or advancing) them, then the side effect would affect other developers. We don't want that!

The only way to handle this would be to create a new project where the R/O components could be set to a previous set. This would be much more intensive and could easily be come unwieldy. For example, if you wanted 3 days worth of "history" we would need to create 3 projects to contain them.