" /> Status for Andrew DeFaria: December 25, 2005 - December 31, 2005 Archives

« December 18, 2005 - December 24, 2005 | Main | January 1, 2006 - January 7, 2006 »

December 30, 2005

pulse

  • Added some functionality to Clearcase::Vob and Clearcase::View
  • Coded a pulse script which performs a pulse checking by checking out and in a file 10 times. If the pulse takes more than 60 seconds then an error is reported

Pulse

I decided to make the pulse script part of a larger project so I created a perf subdirectory under cc. The idea here is that pulse is a sort of performance test - a "sanity performance test" if you will. In general the script does the following:

  • Creates a logfile (pulse.log)
  • Creates a new view (default) if required
  • Sets context to that view
  • Changes directory to the view to the vob (/vobs/adm by default)
  • Performs 10 checkouts and checkins of a relatively large file (a copy of firefox.exe which is ~7 Meg)
  • Compares the time that those checkouts and checkins took against a threshold (default 60) and reports an error if it took too long

Here is a usage:

ccase-rmna-3:export SITE_PERL_LIBPATH=~/SCM/lib
ccase-rmna-3:pulse -u
Usage:  pulse (v1.0) [-u] [-v] [-d] [-view ] [-vob ]
        [-element ] [-t ] [-i ]

Where:

  -u:       Display usage
  -v:       Turn on verbose mode
  -d:       Turn on debug mode
  -view:    View tag to create/use (Default: default)
  -vob:     Vob tag to use (Default /vobs/adm)
  -element: Vob relative path to element to checkout/in (Default: firefox.exe)
  -t :   Threshold of what is "too long" (Default 60 seconds)
  -i :   Number of iterations (default 10)

Here's an example of the log file produced:

pulse: 12/30/2005 @ 12:27: Performing 10 checkout/ins in view default vob /vobs/adm of element firefox.exe
pulse: 12/30/2005 @ 12:28: Finished in 36 seconds
pulse: 12/30/2005 @ 12:28: Performing 10 checkout/ins in view default vob /vobs/adm of element firefox.exe
pulse: 12/30/2005 @ 12:29: Finished in 35 seconds

Several issues came up when attempting to code this script. For example:

  • Which view should this operate in?
  • Which vob should this operate in?
  • How will we assure that this vob exists at any site that we want to run pulse? Perhaps we should make a vob for performance testing only (e.g. /vobs/perf).
  • Checking out and in a single element creates many versions on the version tree. If this pulse script was run every hour then 240 versions would be created in only one day. It would be better to more properly "setup" and "teardown" the whole environment but this would be considerably more complex and time consuming. For example, pulse could create the view, create the vob, formulate some elements and do mkelems on them, do the checkouts and checkins, do rmelems and remove the vob and view, etc. That's a lot! Alternately it could simply do the mkelems, the checkouts and checkins, then do rmelems but that brings up the issue of where does the source come for the mkelems? Also, typically there's a trigger to prevent rmelem...
  • Currently the script is pretty Unix centric. For example, it uses cleartool lsview -stgloc -auto. This works on Unix but for some reason -stgloc doesn't work on Windows. I also suspect that at the various sites usage of -stgloc might not be that reliable.
  • Also the view used is a dynamic view. Should a snapshot version be coded?
  • Current the script uses /vobs/ for a vob tag. In fact it uses /vobs/adm. This is not a Windows vob tag so the script will not work on Windows
  • The script does sport paramaters to change things such as the view used, the vob used and the element to checkout and in. The defaults are great but then again we don't have a performance testing environment setup and replicated yet
  • Currently the script logs it's activity to a log file but it's not in a great format for analysis. The TimeUtils module I'm using only as 1 second resolution. I'm not sure how much analysis of this logfile is desired
  • There is no facility for emailing alerts when the script determines that things are taking too long

December 29, 2005

Clearcase Modules/log_activity

  • Worked on creating several Clearcase modules: Clearcase::Vobs, Clearcase:Vob and Clearcase::View
  • Updated log_activity to use new Clearcase modules

Clearcase Modules

It occurs to me that lots of repeatative Perl coding goes on with many people implementing functions for the same old commonly performed actions. For example, sending an email message, reading a file into an array and performing many Clearcase operations such as creating a view, checking to see if a view exists, mounting vobs, etc. I thought it's about time to have some Perl modules to offer these common tasks in a useful way. In particular, when writing log_activity I had to check to see if a particular view exists and create it if it doesn't, mount all the vobs, etc. Seems to me a set of Clearcase Perl modules would be useful to me, Broadcom and indeed others so I started coding them.

The idea here is to provide Perl objects for Clearcase objects that can be used and manipulated in a Perl manner. So a vob object would contain much of the registry information, in case you need it, as well as provide some common "actions" such a mount the vob represented by this object. The Vobs object would contain information about all vobs and things like itterrators and the like. A mount on a Vobs object would mean mount all vobs, etc. Similarly with a View object one could imagine methods such as exists which tells you if the view exists, create to create a view, remove to remove a view and so on.

Now these are not the only Clearcase oriented "objects" that need to be made into Perl objects nor do the current objects cover all of the access to the various functionality that they could - Clearcase is a huge system! But one must start somewhere...

Similarly I plan on making more modules for other things, like Utils.pm for things like reading in a whole file and returning an array or perhaps a string (wantarray would be useful here). The general idea is to start making more userful modules that are Perl like so that one can more quickly develop code and not slow themselves down reinventing the Perl wheel all the time. Combining this with hosting this code using CVS and my server should prove useful. The Clearcase object(s) may even be something that can be packaged up and perhaps put on CPAN (though I need to learn a lot more about things like POD and other things to make this comply better with any CPAN requirements), but all of this will be a "work in progress" for quite some time though. Remember though - "Every journey begins with the first step"...

December 28, 2005

log_activity/create_dev_snapview.pl

  • Worked on log_activity Perl script
  • Investigated issue with R/O components for create_dev_snapview.pl

log_activity

Ray Tran wrote:

To summarize the three logging projects. The primary log directory is /projects/scm_tools_logs in San Jose. Please let me know if you have any input or suggestions

We need a global area that's truly global!

Turns out that /projects/scm_tools_logs doesn't exist. I assume you meant /projects/IT_SCM/logs. However /projects/IT_SCM doesn't exist on either ccase-rmna-3 nor ccase-sj1-1... So running a script that requires Clearcase such as this I cannot access the directory where I need to deposit the log files! Additionally I cannot access /projects/IT_SCM/SCM/lib for my libraries nor /projects/IT_SCM/SCM/cc where the script would be residing...

Activity logs

This is a log of who checked things out and into ClearCase and from what view. I think we can do this hourly and append to the log file so it doesn't take too long to run.

Turns out lshistory is pretty flexible and efficient. One does not need to do a cleartool find on all elements and then call lshistory on individual elements, etc. The lshistory command supports a -recursive parameter. But we really want to report things across vobs. The lshistory command also supports a -avobs parameter. Two limitations here... First lshistory needs to work the context of a view. Secondly, -avobs only works on vobs that are mounted.

I've coded up a script called log_checkins which will create a view (named default <- I can change this) if necessary and also mounts all vobs prior to calling lshistory. It also supports a -n parameter for number of days - thus -n 7 will produce a weeks worth of log files. Logfiles are named checkin.<date>.log where <date> is the date in YYYYMMDD format. So -n 1 (or not specifying -n) will produce a log file for the previous day (e.g. checkin.20051227.log) encompassing all activity on that day, while specifying -n 7 will produce 7 logfiles, one for each of the last 7 days properly dated.

The fields in the logfile (separated with ";") are:

  1. <date>.<time> (<date> is YYYYMMDD and <time> is HHMMSS 24 hour format). This format makes the logfile easily sortable by date and time with sort(1).
  2. <username> (Not sure where lshistory gets the username, normally is the same as the next field but sometimes it's different - e.g. "Clear Case Administrator")
  3. <user>@<host> The user login name (e.g. adefaria) @ host (e.g. ltsjca-adefaria)
  4. <event> The Clearcase event
  5. <element> View extended path name

Clearcase Events

I ran log_checkins for the last 30 days and examined the log files. No filtering for only "check in" events were done. The range of Clearcase events are:

  • checkout directory version
  • checkout version
  • create activity
  • create baseline
  • create branch
  • create branch type
  • create directory element
  • create directory version
  • create file element
  • create hyperlink
  • create label type
  • create trigger type
  • create version
  • destroy activity in versioned object base
  • destroy checkpoint "<checkpoint>" in versioned object base
  • destroy element in versioned object base
  • destroy hyperlink "<hyperlink> in versioned object base
  • destroy sub-branch "<sub-branch>" of branch
  • destroy type in versioned object base
  • destroy version on branch
  • import file element
  • import hyperlink
  • lock activity
  • lock branch
  • lock branch type
  • lock trigger type
  • lock versioned object base
  • unlock activity
  • unlock versioned object base

There are probably other event types. Also, in most cases the <element> field contains additional information that can be extracted. For example:

20051201.151257;drambo;drambo@PCRMNA-DRAMBO;create
version;/vobs/OnePhone/pub/Makefile@@/main/OnePhone_mainline_Intr/OnePhone_3.2_Intr_new/10

one can see that this was a creation of a version in the OnePhone vob of pub/Makefile. The part after the @@ is the branch information (/main/OnePhone_mainline_Intr/OnePhone_3.2_Intr_new) and version 10.

Although you asked for checked out things, checked in things are more interesting. In fact I'd say that all of the above Clearcase events are interesting and that we should just log all of it (and I'll change log_checkins -> log_activity and change checkin.<date>.log -> activity.<date>.log

Pulse check log

This is a script that should run a quick checkout and time the process. This should run at least every 15 minutes. I'm not sure if we need to set this up for each VOB or if we can just create a IT_SCM VOB and checkout from there. We then control how much data gets moved. For example in Perforce, I've created some file that are 100MB in size. I total the files and check this out. I now have control over the data.

Statistic logs

This is a script that would create two logs - one general data for each site, one for each VOB of each site.

The general data should have at the least - date_time;site;number of VOBs; number of total VOB size; number of snapshot views; number of dynamic views; number of file elements; number of directory element;...

The VOB specific data - date_time;site;VOB_name;size of database;size of source pool; size of devired object; size of cleartext; size of admin data

The directory structure that I envisioned is as such

/projects/scm_tools_logs
 
 - cc
      - 2005
          - 12
             - 01
 

I see little benefit in constructing such a directory structure, especially programmatically. Instead I place the YYYYMMDD in the file name of the log file in a format described above. This format is easily parseable, sortable and grepable and it also compresses nicely. A directory structure such as above could easily be created at a later time or sets of logfiles could be easily gzipped and tar'ed (e.g. mkdir 2005; tar -zcf 2005/activity.logs.gz activity.2005*; rm activity.2005*).

R/O Components & create_dev_snapview.pl

After speaking at length with Shivdutt about this and learning more about UCM, projects, streams and the like I see a problem here. Let me explain it based on how I see this...

With UCM you have projects which contain components which can be R/W or R/O. Projects also have streams, one integration stream (and an associated integration view) and one or more development streams (and their associated development views).

When a development view is created in a development stream it is populated with all of the components in the project based on baselines. The difference here is that you can use different baselines for the various R/W components but you cannot use different baselines for the R/O components. The baseline information for R/O components is kept "globally" at the project level and is not modifiable in the development streams. Since, theoretically, R/O components are not modifiable they considered shared for all development streams.

While R/O components can have their baselines changed (usually advanced) at the project level, the act of doing so affects all development streams (dynamic development streams will be affected immediately while snapshot development views at next update time).

So if we had create_dev_snapview.pl alter the baselines of the R/O components by retarding (or advancing) them, then the side effect would affect other developers. We don't want that!

The only way to handle this would be to create a new project where the R/O components could be set to a previous set. This would be much more intensive and could easily be come unwieldy. For example, if you wanted 3 days worth of "history" we would need to create 3 projects to contain them.

December 27, 2005

log_checkins

  • Started investigating a log_checkins procedure
  • Worked on Andrew's create_dev_snapview.pl