Status for Andrew DeFaria

• Added some functionality to Clearcase::Vob and Clearcase::View
• Coded a pulse script which performs a pulse checking by checking out and in a file 10 times. If the pulse takes more than 60 seconds then an error is reported

Pulse

I decided to make the pulse script part of a larger project so I created a perf subdirectory under cc. The idea here is that pulse is a sort of performance test - a "sanity performance test" if you will. In general the script does the following:

• Creates a logfile (pulse.log)
• Creates a new view (default) if required
• Sets context to that view
• Changes directory to the view to the vob (/vobs/adm by default)
• Performs 10 checkouts and checkins of a relatively large file (a copy of firefox.exe which is ~7 Meg)
• Compares the time that those checkouts and checkins took against a threshold (default 60) and reports an error if it took too long

Here is a usage:

ccase-rmna-3:export SITE_PERL_LIBPATH=~/SCM/lib
ccase-rmna-3:pulse -u
Usage:  pulse (v1.0) [-u] [-v] [-d] [-view ] [-vob ]
        [-element ] [-t ] [-i ]

Where:

  -u:       Display usage
  -v:       Turn on verbose mode
  -d:       Turn on debug mode
  -view:    View tag to create/use (Default: default)
  -vob:     Vob tag to use (Default /vobs/adm)
  -element: Vob relative path to element to checkout/in (Default: firefox.exe)
  -t :   Threshold of what is "too long" (Default 60 seconds)
  -i :   Number of iterations (default 10)

Here's an example of the log file produced:

pulse: 12/30/2005 @ 12:27: Performing 10 checkout/ins in view default vob /vobs/adm of element firefox.exe
pulse: 12/30/2005 @ 12:28: Finished in 36 seconds
pulse: 12/30/2005 @ 12:28: Performing 10 checkout/ins in view default vob /vobs/adm of element firefox.exe
pulse: 12/30/2005 @ 12:29: Finished in 35 seconds

Several issues came up when attempting to code this script. For example:

• Which view should this operate in?
• Which vob should this operate in?
• How will we assure that this vob exists at any site that we want to run pulse? Perhaps we should make a vob for performance testing only (e.g. /vobs/perf).
• Checking out and in a single element creates many versions on the version tree. If this pulse script was run every hour then 240 versions would be created in only one day. It would be better to more properly "setup" and "teardown" the whole environment but this would be considerably more complex and time consuming. For example, pulse could create the view, create the vob, formulate some elements and do mkelems on them, do the checkouts and checkins, do rmelems and remove the vob and view, etc. That's a lot! Alternately it could simply do the mkelems, the checkouts and checkins, then do rmelems but that brings up the issue of where does the source come for the mkelems? Also, typically there's a trigger to prevent rmelem...
• Currently the script is pretty Unix centric. For example, it uses cleartool lsview -stgloc -auto. This works on Unix but for some reason -stgloc doesn't work on Windows. I also suspect that at the various sites usage of -stgloc might not be that reliable.
• Also the view used is a dynamic view. Should a snapshot version be coded?
• Current the script uses /vobs/ for a vob tag. In fact it uses /vobs/adm. This is not a Windows vob tag so the script will not work on Windows
• The script does sport paramaters to change things such as the view used, the vob used and the element to checkout and in. The defaults are great but then again we don't have a performance testing environment setup and replicated yet
• Currently the script logs it's activity to a log file but it's not in a great format for analysis. The TimeUtils module I'm using only as 1 second resolution. I'm not sure how much analysis of this logfile is desired
• There is no facility for emailing alerts when the script determines that things are taking too long

Posted by at 10:57 AM | Permalink

• Worked on creating several Clearcase modules: Clearcase::Vobs, Clearcase:Vob and Clearcase::View
• Updated log_activity to use new Clearcase modules

Clearcase Modules

It occurs to me that lots of repeatative Perl coding goes on with many people implementing functions for the same old commonly performed actions. For example, sending an email message, reading a file into an array and performing many Clearcase operations such as creating a view, checking to see if a view exists, mounting vobs, etc. I thought it's about time to have some Perl modules to offer these common tasks in a useful way. In particular, when writing log_activity I had to check to see if a particular view exists and create it if it doesn't, mount all the vobs, etc. Seems to me a set of Clearcase Perl modules would be useful to me, Broadcom and indeed others so I started coding them.

The idea here is to provide Perl objects for Clearcase objects that can be used and manipulated in a Perl manner. So a vob object would contain much of the registry information, in case you need it, as well as provide some common "actions" such a mount the vob represented by this object. The Vobs object would contain information about all vobs and things like itterrators and the like. A mount on a Vobs object would mean mount all vobs, etc. Similarly with a View object one could imagine methods such as exists which tells you if the view exists, create to create a view, remove to remove a view and so on.

Now these are not the only Clearcase oriented "objects" that need to be made into Perl objects nor do the current objects cover all of the access to the various functionality that they could - Clearcase is a huge system! But one must start somewhere...

Similarly I plan on making more modules for other things, like Utils.pm for things like reading in a whole file and returning an array or perhaps a string (wantarray would be useful here). The general idea is to start making more userful modules that are Perl like so that one can more quickly develop code and not slow themselves down reinventing the Perl wheel all the time. Combining this with hosting this code using CVS and my server should prove useful. The Clearcase object(s) may even be something that can be packaged up and perhaps put on CPAN (though I need to learn a lot more about things like POD and other things to make this comply better with any CPAN requirements), but all of this will be a "work in progress" for quite some time though. Remember though - "Every journey begins with the first step"...

Posted by at 6:48 PM | Permalink

• Worked on log_activity Perl script
• Investigated issue with R/O components for create_dev_snapview.pl

log_activity

Ray Tran wrote:

To summarize the three logging projects. The primary log directory is /projects/scm_tools_logs in San Jose. Please let me know if you have any input or suggestions

We need a global area that's truly global!

Turns out that /projects/scm_tools_logs doesn't exist. I assume you meant /projects/IT_SCM/logs. However /projects/IT_SCM doesn't exist on either ccase-rmna-3 nor ccase-sj1-1... So running a script that requires Clearcase such as this I cannot access the directory where I need to deposit the log files! Additionally I cannot access /projects/IT_SCM/SCM/lib for my libraries nor /projects/IT_SCM/SCM/cc where the script would be residing...

Activity logs

This is a log of who checked things out and into ClearCase and from what view. I think we can do this hourly and append to the log file so it doesn't take too long to run.

Turns out lshistory is pretty flexible and efficient. One does not need to do a cleartool find on all elements and then call lshistory on individual elements, etc. The lshistory command supports a -recursive parameter. But we really want to report things across vobs. The lshistory command also supports a -avobs parameter. Two limitations here... First lshistory needs to work the context of a view. Secondly, -avobs only works on vobs that are mounted.

I've coded up a script called log_checkins which will create a view (named default <- I can change this) if necessary and also mounts all vobs prior to calling lshistory. It also supports a -n parameter for number of days - thus -n 7 will produce a weeks worth of log files. Logfiles are named checkin.<date>.log where <date> is the date in YYYYMMDD format. So -n 1 (or not specifying -n) will produce a log file for the previous day (e.g. checkin.20051227.log) encompassing all activity on that day, while specifying -n 7 will produce 7 logfiles, one for each of the last 7 days properly dated.

The fields in the logfile (separated with ";") are:

1. <date>.<time> (<date> is YYYYMMDD and <time> is HHMMSS 24 hour format). This format makes the logfile easily sortable by date and time with sort(1).
2. <username> (Not sure where lshistory gets the username, normally is the same as the next field but sometimes it's different - e.g. "Clear Case Administrator")
3. <user>@<host> The user login name (e.g. adefaria) @ host (e.g. ltsjca-adefaria)
4. <event> The Clearcase event
5. <element> View extended path name

Clearcase Events

I ran log_checkins for the last 30 days and examined the log files. No filtering for only "check in" events were done. The range of Clearcase events are:

• checkout directory version
• checkout version
• create activity
• create baseline
• create branch
• create branch type
• create directory element
• create directory version
• create file element
• create hyperlink
• create label type
• create trigger type
• create version
• destroy activity in versioned object base
• destroy checkpoint "<checkpoint>" in versioned object base
• destroy element in versioned object base
• destroy hyperlink "<hyperlink> in versioned object base
• destroy sub-branch "<sub-branch>" of branch
• destroy type in versioned object base
• destroy version on branch
• import file element
• import hyperlink
• lock activity
• lock branch
• lock branch type
• lock trigger type
• lock versioned object base
• unlock activity
• unlock versioned object base

There are probably other event types. Also, in most cases the <element> field contains additional information that can be extracted. For example:

20051201.151257;drambo;drambo@PCRMNA-DRAMBO;create
version;/vobs/OnePhone/pub/Makefile@@/main/OnePhone_mainline_Intr/OnePhone_3.2_Intr_new/10

one can see that this was a creation of a version in the OnePhone vob of pub/Makefile. The part after the @@ is the branch information (/main/OnePhone_mainline_Intr/OnePhone_3.2_Intr_new) and version 10.

Although you asked for checked out things, checked in things are more interesting. In fact I'd say that all of the above Clearcase events are interesting and that we should just log all of it (and I'll change log_checkins -> log_activity and change checkin.<date>.log -> activity.<date>.log

Pulse check log

This is a script that should run a quick checkout and time the process. This should run at least every 15 minutes. I'm not sure if we need to set this up for each VOB or if we can just create a IT_SCM VOB and checkout from there. We then control how much data gets moved. For example in Perforce, I've created some file that are 100MB in size. I total the files and check this out. I now have control over the data.

Statistic logs

This is a script that would create two logs - one general data for each site, one for each VOB of each site.

The general data should have at the least - date_time;site;number of VOBs; number of total VOB size; number of snapshot views; number of dynamic views; number of file elements; number of directory element;...

The VOB specific data - date_time;site;VOB_name;size of database;size of source pool; size of devired object; size of cleartext; size of admin data

The directory structure that I envisioned is as such

/projects/scm_tools_logs
 
 - cc
      - 2005
          - 12
             - 01
 

I see little benefit in constructing such a directory structure, especially programmatically. Instead I place the YYYYMMDD in the file name of the log file in a format described above. This format is easily parseable, sortable and grepable and it also compresses nicely. A directory structure such as above could easily be created at a later time or sets of logfiles could be easily gzipped and tar'ed (e.g. mkdir 2005; tar -zcf 2005/activity.logs.gz activity.2005*; rm activity.2005*).

R/O Components & create_dev_snapview.pl

After speaking at length with Shivdutt about this and learning more about UCM, projects, streams and the like I see a problem here. Let me explain it based on how I see this...

With UCM you have projects which contain components which can be R/W or R/O. Projects also have streams, one integration stream (and an associated integration view) and one or more development streams (and their associated development views).

When a development view is created in a development stream it is populated with all of the components in the project based on baselines. The difference here is that you can use different baselines for the various R/W components but you cannot use different baselines for the R/O components. The baseline information for R/O components is kept "globally" at the project level and is not modifiable in the development streams. Since, theoretically, R/O components are not modifiable they considered shared for all development streams.

While R/O components can have their baselines changed (usually advanced) at the project level, the act of doing so affects all development streams (dynamic development streams will be affected immediately while snapshot development views at next update time).

So if we had create_dev_snapview.pl alter the baselines of the R/O components by retarding (or advancing) them, then the side effect would affect other developers. We don't want that!

The only way to handle this would be to create a new project where the R/O components could be set to a previous set. This would be much more intensive and could easily be come unwieldy. For example, if you wanted 3 days worth of "history" we would need to create 3 projects to contain them.

Posted by at 12:59 PM | Permalink

• Started investigating a log_checkins procedure
• Worked on Andrew's create_dev_snapview.pl

Posted by at 12:56 PM | Permalink

• Gave demonstration of Clearquest to Jene Fan and her group. Went pretty well
• Investigating issue where cvsims seems to be called when one cvs adds a directory

Posted by at 2:39 PM | Permalink

• Set up a test bugs database for a demo
• Set up p4test as a temp server for PQA on production data
• Further coding of cvsims and integration into CVS

CVSIMS Issues

There are still some issues WRT cvsims:

• When you cvs add a directory it seems that cvsims is called. There is no issue IDs and thus thsi is a problem
• Currently the path to the file being committed is not being passed. There is a %p that can be added to the command line in loginfo but it merely says "path relative to repository". What does that mean when you are checking in foo/foo.h, foo/foo.c and bar/bar.h?

Turns out that cvs will make calls per path. IOW if you are checking in foo/foo.h, foo/foo.c and bar/bar.h cvs would call the loginfo trigger with path = foo for foo.h and foo.c then call the loginfo trigger again with path = bar for bar.h. Unfortuantely this would leave us with the more ugly:

### adefaria Thu Dec 22 19:14:09 2005 ###
1.6 /projects/ccase_irva/cvsmaster/ims/foo/foo.h
1.7 /projects/ccase_irva/cvsmaster/ims/foo/foo.c

### adefaria Thu Dec 22 19:14:10 2005 ###
1.4 /projects/ccase_irva/cvsmaster/ims/bar/bar.h

As you can see above I added both a -path and -repository options so I can compose the full path to each file.

One thing that is still not handled is the concept of branches. I'm not sure if that is required...

Posted by at 6:55 PM | Permalink

• Created test bugs database for Clearquest evaluation
• Worked on integrating cvsims into CVS
• PQA web problems are surfacing. Need to set up newer web server

Posted by at 6:43 PM | Permalink

• Changed cvsims to use new global area
• Created document for CVS/IMS Connector
• Created new global area
• Working on another users problems with accessing ccase-rmna-4

Global File System Area - SCM

I'm just trying to understand a little bit better about global paths, replication and DFS here at Broadcom. I ask because I see that this new SCM directory that I created over in Irvine is not at other sites as I would have expected it. For example:

ccase-rmna-3:ls -d /projects/ccase/SCM
ls: /projects/ccase/SCM: No such file or directory
ccase-rmna-3:ls /projects/ccase | wc -l
     17

and

ccase-sj1-1:ls -d /projects/ccase/SCM
ls: /projects/ccase/SCM: No such file or directory
ccase-sj1-1:ls /projects/ccase | wc -l
     16

However

x-irva-01:ls -d /projects/ccase/SCM                                           
/projects/ccase/SCM/
x-irva-01:ls /projects/ccase | wc -l                                          
      24

So then the SCM directory exists in Irvine but not in SJ nor Richmond. I suspect that it's merely that something needs to replicate it and maybe I just need to wait until that happens. Still such a delay in replication is a side effect we must bear in mind when we are pushing out changes... Also, notice the different directory counts at the three sites above...

WRT Windows and DFS, my Z drive is mapped to \\brcm-sj\dfs and I therefore do not have a Z:\projects\ccase\SCM directory there either:

Local:ls -d /dev/z/projects/ccase/SCM
ls: /dev/z/projects/ccase/SCM: No such file or directory
Local:ls /dev/z/projects/ccase | wc -l
16

I suspect this correlates to ccase-sj1-1:/projects/ccase/SCM and whenever that directory gets populated so will my Z drive.

Again, it might be that replication will eventually happen or that it need to be set up. But we must understand that pushing something to some "global" place in Irvine, SJ or Richmond has a built in prorogation delay.

Finally, it seems to me that the intent of the projects automount map is to provide a well known path to global file space that exists on filers at the various sites and that replication between the filers happens on some known schedule. Scripts or other processes should "reach out" to the filer storage area they they know (e.g. \\brcm-irv\dfs\projects\ccase...) rather they should use the automount map (i.e. /projects/ccase) so as to avoid having to go out on the WAN to get stuff. For Unix this is pretty simplified in that the automounter and the projects automount map normalizes this to just /projects//. But can it be said that the same is true under Windows for the Z drive and DFS? By that I mean can I rely on Z:\projects\ccase\SCM to be the right place? If so then I will change my Perl code to set $lib_path = "Z:\\projects\\ccase\\SCM\\lib".

Posted by at 5:27 PM | Permalink

• BinRebase may need to use findmerge instead of rebase
• Worked with Shivdutt re: satellite San Jose/Richmond user

BinRebase may need to use findmerge

Rebasing is different than delivering. With Inter Project Deliver developers are "delivering" from one project to the next. This is not a real delivery as in delivering to the integration stream, rather it's a delivery from one project to another. What we have here is a hierarchy outside of the normal heirarchy supported by UCM.

Delivery between projects therefore is merely a findmerge and indeed the deliver command is not used. Rebasing, called Rebase to parent, however still uses the rebase command. This presents a problem in that rebase -preview -long does not operate the same way as findmerge -print. For one it does not do recursive directory merging like findmerge does (with findmerge we can capture and control the merging by using -print and handling the merging by ourselves). IBM/Rational designed deliver and rebase to be as automatic as possible. I suspect therefore that rebase automatically handles recursive directory merging thus not allowing you to do the recursive directory merging yourself.

"So fine" you say, "Just let rebase handle all of the directory merging". Well there is no way to do just directory merging and remember we want to capture and handle the file element merging ourselves so we can capture those instances where merge conflicts and binary merges exist, delaying them to the very end and handling the binary merge (or rebase) ourselves. With rebase we don't seem to have the same options as with findmerge and thus this may not be doable.

"OK, fine" you say again, "Skip the rebasing and just use findmerge". While that may be doable we will not have the "rebase in progress" behavior that they are currently accustomed to. That plus rebase opperates off of baselines and even recommendated baselines. We would need to figure out how that translates to labels that findmerge can use - and they'd need to be labels of possibly multiple components and the like. For now we are shelfing this BinRebase until we obtain more input...

Posted by at 5:09 PM | Permalink

• Set up GNATS access for a user
• Helped Vinh with issues regarding old CQ Web and a bug that it has
• Assisted a few users with resolving Evil Twin problems
• More fully documented how to resolve Evil Twin problems by merging the parent directory
• Investigated implementing the other half of BinMerge where we use rebase instead of find merge

Rebase

Initially I thought that rebase did not work like findmerge in that it didn't have options for performing essentially the merge operate and just telling you what needed to merge. Turns out it does have that option - cleartool merge -preview -long. The long option causes rebase to list out what needs to be merged.

But rebase is more than just a findmerge. Additionally one needs to do the following:

• Create and set an activity for the view
• Check out items first
• Perform the merge with -abort noting which merges failed and how they failed
• Execute the rebase command to mark the view as being rebased
• Execute the rebase -complete to mark the rebase completed (IFF all elements are rebased)

Of course this is just a preliminary list - we still need to figure out if rebasing, merging actually, of directories need to be done first (probably), how to execute rebase to mark "rebase in progress" without actually attempting to merge anything, etc.

Posted by at 5:06 PM | Permalink

• Found and fixed bug in get_views_for_stream
• Fixed bug in create_dev_snapview.pl::ParseBaselines

get_views_for_stream

Hi Naga. I'm writing you this because I believe there's a bug, possibly with Clearcase, but that seems to manifest itself in code I believe you wrote. I came across this implementing enhancements to create_dev_snapview.pl for Andrew Feltham. In essence Andrew wants to be able to have multiple views in a stream. I've implemented a -reuse_stream parm to allow this. If -reuse_stream is used then the code will skip recreating the stream and effectively just make another view in that same stream. If -reuse_stream is not specified then the stream is removed - along with any views that that stream has.

It appears as if the code was coded to attempt to handle the case where a stream has multiple views however it fails due to a possible but with lsstream. When calling create_stream the code checks to see if the stream exists and if it does it attempts to recreate the stream by determining the view(s) in the stream, deleting them, deleting the stream and creating the stream anew.

To determine the view(s) in the stream get_views_for_stream is called. Here's the original function:

sub get_views_for_stream {
    my $sel_pvob = $_[0];
    my $sel_stream = $_[1];

    my @views = split(/\n/, `cleartool lsstream -fmt \"\%\[views\]p\\n\" stream:$sel_stream\@\\$sel_pvob`);
    return @views;
}

Essentially it does an lsstream asking for only the views, separated by newlines. The problem is that lsstream (with that format) does not return the views separated by newlines:

P:\>cleartool lsstream -fmt "%[views]p\n" stream:Build_rmnaNT_adefaria_LTSJCA-ADEFARIA_sv_ldx_2.3@\rmna_projects
adefaria_test adefaria_test2 adefaria_test3

P:\>

As you can see adefaria_test, adefaria_test2 and adefaria_test3 all appear on the same line. The code then splits the line on \n and effectively always returns a 1 entry array with possibly multiple view names in the first entry separated by spaces. Needless to say this will fail later on when remove_view is called and it generates a "cleartool rmview -force <view names>".

Here's my proposed fix:

sub get_views_for_stream {
    my $sel_pvob = $_[0];
    my $sel_stream = $_[1];
    # Modified Wed Dec 14 11:25:09 PST 2005 Andrew@DeFaria.com
    #
    # Note: The format -fmt "%[views]p\n" below does not work! What is
    # expected to be returned is:
    #
    # view1
    # view2
    # view3
    #
    # However what is actually returned is:
    #
    # view1 view2 view3
    #
    # Therefore the old spilt of /\n/ always returned a single entry
    # array with possibly multiple view names in the first
    # entry. Changed to use / / for split and not append a \n.
    my @views = split(/ /, `cleartool lsstream -fmt \"\%\[views\]p\" stream:$sel_stream\@\\$sel_pvob`);
    return @views;
}

Is this acceptable to you?

Posted by at 1:48 PM | Permalink

• Updated triggers on ccase-rmna-3 as some new vobs seem to lack them
• Updated cvsims to parse input and handle both -pre and -post operations
• Had problems with CVS admin files

CVSIMS

I have managed to learn how to update IMS and I'm at the point where I need to understand the input requirements better. I will describe what I envision happening and follow it with a few questions.

In general the desire seems to be to update IMS when a commit happens in certain CVS repositories¹. As such commit comment information will need to conform to a certain format. Commit comments need to have the following in them identifying the IMS Issue ID they are associated with:

IMSIssueID: <issue ID>

Pre-commit checks (i.e. verifymsg)

The string "IMSIssueID: " must appear at the start of a line followed by the issue ID². The IMSIssueID must exist in IMS³. If the issue ID is not in IMS then the commit will not be allowed. Commits lacking any IMSIssueID will be rejected⁴.

Post-commit operations (i.e. loginfo)

If the pre-commit checks pass the IMS issue will have it's change set field updated with the file names of the committed files. As change set is an append only field the following separator will be added first followed by the list of files⁵:

# <USERNAME> <DATE_TIME> committed:
file1
file2
...

Again, as change set is an append only field there may be multiple updates like so:

# <USERNAME> <DATE_TIME> committed:
file1
file2
...
# <USERNAME> <DATE_TIME> committed:
file10

Questions

1. Which repositories does this apply?
2. Are multiple IMSIssueIDs allowed? If so then are multiple lines such as the above acceptable or should the script search for perhaps a comma separated list of issue IDs?
3. Should the script also check to insure that the issue ID is not in say a closed state?
4. Is it true that all commits in a repository/module/directory must be associated to an IMS Issue?
5. In addition to the file name should we include the revision checked in? Also should any other information be logged (e.g. repository)?

CVS Admin Files

I got myself into a pickle. Here's what I was doing...

I wanted to modify rcsinfo, verifymsg and loginfo in our CVS test area (:pserver:adefaria@cvsps-irva-1:/projects/ccase_irva/cvsmaster) so I checked out CVSROOT and modified those files. The idea was to have rcsinfo point to a template file so the user knows where to put the IMSIssueID, change verifymsg to call cvsism (the connector) to verify that it can find the IssueID and that the IssueID exists in IMS and change loginfo to call cvsism to update the change set after the commit. Unfortunately I modified the files in such a way that I cannot check in at all. I need help in fixing this because I believe what needs to be done needs to be done on the server and I don't have a login there let alone root access.

The problem is that I don't understand what the cwd is when CVS fires off these triggers. For example, in rcsinfo I have the following line:

ALL /tmp/IMSIssueTemplate

"ALL" means for all files and /tmp/IMSIssueTemplate is the template file. But where is /tmp? Is it supposed to be on the server? Where's a good place to put these template files?

In verifymsg I have:

DEFAULT SCM/cvsbin/cvsims -pre

I want to run cvsims (which is in my home directory under SCM/cvsbin) but this also fails. As a result of this I cannot use cvs commit to change verifymsg, hence I'm stuck!

In loginfo I have:

DEFAULT SCM/cvsbin/cvsims -post %{s}

Again SCM/cvsbin/cvsims is not a valid path.

How can I get this fixed? Where should I put these files?

Posted by at 12:35 PM | Permalink

• Worked with Shivdutt on clearfsimport
• Updated mktriggers to use fully qualified paths to Perl
• Added stuff to CVS area - have not committed yet - wanna establish a commitinfo script to get an idea of what input is available to commit

Posted by at 6:14 PM | Permalink

• Changed stripmime to perform logging if -v is present. Also debugged stripmime to the point where it seems to be properly being passed off the queue-pr
• Implemented IMS.pm - a more Perl-like interface to IMS. Also started implementing cvs_ims to be a connector between CVS <-> IMS

Posted by at 4:43 PM | Permalink

• Worked with David Dinh on getting Clearquest Windows Client on Citrix
• Updated BinMerge to always draw the merge arrow to the branch we are merging to
• Fixed BinMerge's PerlTk portion to top the window iniitially
• Worked with Ravi on Stripmime and GNATS

BinMerge updates

Updated BinMerge to always draw the merge arrow to the branch we are merging to. There was also a bug about the choice returned from the PerlTk portion where it needed to not only be chomped but chopped until there were no more "\n"'s there. Don't know why that happened.

Additionally the PerlTk window would often not pop to the top of the window stack. This is apparenty common under Windows. I implemented the following code:

    # Make sure the window pops to the top
    # Trying really hard... :-)
    $main->update;
    $main->deiconify;
    $main->update;
    $main->raise;
    $main->update;
    $main->focusForce;
    $main->update;

    MainLoop;

At least this heavy handed approach seems to work.

Stripmime and GNATS

Ravi and I worked on the stripmimed thing a little today. Here's what happened:

• We verified that stripmimed is indeed running. The alias was changed to not run Perl first rather to use the shebang line. That was working - IOW /tools/gnats/4.0/bin/stripmimed would run. (later it broke).
• We verified that stripmimed was writing properly to stdout.
• We saw that /tools/gnats/4.0/SunOS/libexec/gnats/queue-pr was being run and a file was being deposited into /tools/gnats/db-test-it/gnats-queue.
• We saw that /tools/gnats/4.0/SunOS/libexec/gnats/queue-pr would be run by cron (or by hand) and it would process the file deposited in the queue
• After that we don't know where the information for this went. We were using Test/1 in the subject line which should update http://gnats-irva-3.broadcom.com/cgi-bin/gnatsweb.pl?debug=&database=IT-Test&cmd=edit&cmd=edit&pr=1 but it isn't.

I attempted to debug this further and I made a change /tools/gnats/4.0/bin/stripmimed and now:

gnats-irva-3:/tools/gnats/4.0/bin/stripmimed
/tools/gnats/4.0/bin/stripmimed: line 44: use: command not found
/tools/gnats/4.0/bin/stripmimed: line 48: use: command not found
/tools/gnats/4.0/bin/stripmimed: line 49: use: command not found
/tools/gnats/4.0/bin/stripmimed: line 50: use: command not found
/tools/gnats/4.0/bin/stripmimed: line 51: use: command not found
/tools/gnats/4.0/bin/stripmimed: line 52: use: command not found
/tools/gnats/4.0/bin/stripmimed: line 54: undef: command not found
/tools/gnats/4.0/bin/stripmimed: line 56: my: command not found
/tools/gnats/4.0/bin/stripmimed: line 57: my: command not found
/tools/gnats/4.0/bin/stripmimed: line 58: my: command not found
/tools/gnats/4.0/bin/stripmimed: line 60: open: command not found
/tools/gnats/4.0/bin/stripmimed: line 61: or: command not found
/tools/gnats/4.0/bin/stripmimed: line 63: sub: command not found
/tools/gnats/4.0/bin/stripmimed: line 64: my: command not found
/tools/gnats/4.0/bin/stripmimed: line 66: print: command not found
/tools/gnats/4.0/bin/stripmimed: line 67: syntax error near unexpected token `}'
/tools/gnats/4.0/bin/stripmimed: line 67: `} # Log'

So how did we fix the shebang problem before? Also, why is queue-pr not properly updating the pr? (My suspicion is that the file attachment on this PR is messing up queue-pr - nope, that doesn't seem to be it...

Posted by at 6:35 PM | Permalink

• Changed bin_merge to be a module so that it easier to call from UCMCustom
• Integrated new BinMerge.pm module into the UCMCustom
• Added fix to use "\" in get_vob_str
• Changed UCMCustom to use strict and warnings(we should always use these)
• Changed UCMCustom to use use instead of require
• Hunted down bug where UCMCustom undef's $/ (Bad UCMCustom!! Bad!)

BinMerge Module

I decided to change bin_merge into a module (BinMerge) and use it in UCMCustom. I was having problems trying to start the bin_merge process and getting it going in a pipe so the output from verbose could give the user feed back. For some reason bin_merge has to exit before any output was returned. Might have been because of the $/ thing described below (Bad UCMCustom, bad!)

Using strict

I've been taught that both use strict and use warnings should always be used when coding Perl. These diagnoistic routines are helpful in catching some common Perl mistakes that are otherwise much more difficult to debug. In fact, in implementing this I found a few potential bugs that I also corrected.

Note: Some version of Perl do not have use warnings. Instead specify -w on the #! line.

Use vs Require

Require is old hat. Use is the new require! From Perldoc's What's the difference between require and use?:

What's the difference between require and use?

Perl offers several different ways to include code from one file into another. Here are the deltas between the various inclusion constructs:

1. do $file is like eval `cat $file`, except the former
1. searches @INC and updates %INC.
2. bequeaths an *unrelated* lexical scope on the eval'ed code.
2. require $file is like do $file, except the former
1. checks for redundant loading, skipping already loaded files.
2. raises an exception on failure to find, compile, or execute $file.
3. require Module is like require "Module.pm", except the former
1. translates each "::" into your system's directory separator.
2. primes the parser to disambiguate class Module as an indirect object.
4. use Module is like require Module, except the former
1. loads the module at compile time, not run-time.
2. imports symbols and semantics from that package to the current one.

Note that becuase use loads the module at compile time one need to arrange for the @INC array to have any paths needed for use to succeed (IOW user written modules). In order to accomplish this one need to seed the @INC arrray in a BEGIN block as BEGIN blocks are executed before use is expanded,

$/ Bug

From Perldoc's Perlvar:

You should be very careful when modifying the default values of most special variables described in this document. In most cases you want to localize these variables before changing them, since if you don't, the change may affect other modules which rely on the default values of the special variables that you have changed. This is one of the correct ways to read the whole file at once:

  open my $fh, "foo" or die $!;
  local $/; # enable localized slurp mode
  my $content = <$fh>;
  close $fh;
  

But the following code is quite bad:

  open my $fh, "foo" or die $!;
  undef $/; # enable slurp mode
  my $content = <$fh>;
  close $fh;
  

since some other module, may want to read data from some file in the default "line mode", so if the code we have just presented has been executed, the global value of $/ is now changed for any other code running inside the same Perl interpreter.

Posted by at 5:44 PM | Permalink

• Fixed stripmime
• Worked with Shivdutt regarding the cleardiffmrg problem. Turns out to be a leading "/". Shivdutt fixed UCMCustom.pl
• Turned off character set validation on p4test for PQA. Have not implemented this on production yet

Stripmime

After banging my head against stripmime all day I finally figured it out. It would work from the command line but not from sendmail! It turned out to be the fact that I was not setting output_dir rather deferring to cwd and in the context of sendmail and aliases I guess that won't do. Changed stripmime to call $parser->output_dir ("/tmp"). Ravi needs to test this then we can implement.

Cleardiffmrg

Tommy Kee was having problems with merging where the cleardiffmrg tool would not come up stating instead that it encountered and error. I had seen this sort of error before at Ameriquest but it was specifically with snapshot views and with views that were tagged with forward slashes (something that mktag allows but cleardiffmrg has a problem with). But here we were dealing with dynamic views and the view tags were with all backslashes (\).

However, UCMCustom.pl would call cleartool findmerge with a leading forward slash (e.g. /andy\path\to\search). I suspected that this leading slash might be causing the problem and eventual showed that it was indeed the problem. Shivdutt changed UCMCustom.pl to use a leading blackslash.

CharacterSetValidation

I called IBM/Rational support this morning to raise the priority on this one. The priority has been raised and IBM/Rational is investigating this more fully. Meantime they suggested, as a temporary workaround, to turn off checking of code pages. I've tried this:

    $ installutil setdbcodepagetonochecking -dbset 2005.02.00 admin 
    *********************************************************
    Starting test setdbcodepagetonochecking
    *********************************************************
    Note: setting the code page to 'no checking' can result
    in data corruption. Please refer to the ClearQuest Release
    Notes for more information.
    Successfully set the code page to 'no checking'.

    *********************************************************
    Exit code 0 for test setdbcodepagetonochecking
    *********************************************************

Note that this does not stop the CharacterSetValidation package from complaining! Turning off the CharacterSetValidation requires a schema change to return FALSE from enable_char_set_validation.

Posted by at 5:20 PM | Permalink

• Told Andrew about new -reuse_stream option to create_dev_snapview.pl. I have him testing this
• Worked with Shivdutt and IBM/Rational on problem with cleardiffmrg. I think I've worked this out to the leading "/" that UCMCustom users.
• Finished stripmime - Ray testing it - created stripmimed that logs to /tmp.
• Worked at getting bin_merge integrated into UCMCustom.pl
• Implemented quick script (topper) to try to catch the hanging problem with ccase-rmna-3

Cleardiffmgr problem

The problem occurs during a deliver between projects where there is a merge conflict and UCMCustom wants to have findmerge call up a Cleardiffmgr to resolve the conflicts. I've seen this before WRT snapshot views and view tags with forward slashes at Ameriquest. UCMCustom perfaces' the vob tags with "/".

But this problem doesn't always happen - in fact most of the time it appears to work. It might be just when there's a merge conflict

After working with Shivdutt and IBM/Rational on the problem and getting down to reproducing it with a long cleartool findmerge command in the user's environment this problem started happening to me in my test environment as I was integrating bin_merge. After much narrowing down I found:

Here's some more information. As you know I started having a similar problem as the end user. I paired it down to the following. This fails (but does not always fail):

    Z:\andys_pvob>cleartool findmerge /andy\bin\cygwin_setup -fver \main\Andrew_Integration\6 -log NUL -gmerge -nc
    Needs Merge "/andy\bin\cygwin_setup" [to \main\Andrew_Integration\Andrew2_Integration\CHECKEDOUT from \main\Andrew_Integration\6 base \main\Andrew_Integration\5]

However the following does work:

    Z:\andys_pvob>cleartool findmerge \andy\bin\cygwin_setup -fver \main\Andrew_Integration\6 -log NUL -gmerge -nc
    Needs Merge "\andy\bin\cygwin_setup" [to \main\Andrew_Integration\Andrew2_Integration\CHECKEDOUT from \main\Andrew_Integration\6 base \main\Andrew_Integration\5]

Notice the difference? It's subtle. The first findmerge's path starts with a "/" while the second one starts with a "\".

bin_merge integration

My bin_merge script remains it's own process. This will be useful if others wish to use this process. However UCMCustom needs to incorporate calling bin_merge. I'm currently having a problem with this because I want to start bin_merge as a pipe so I can give the user feedback. But bin_merge runs to completion before the first output is received. I don't know why this is happening as it doesn't happen in a smaller test.pl environment, only under the massive UCMCustom.pl...

Posted by at 7:54 PM | Permalink

• Researched further code page issues
• Added Everest (Isreal0 team members to PQA
• Added -reuse_stream parm to create_dev_snapview.pl
• Handed a preliminary copy of bin_merge to Jennifer
• Submitted defect to IBM/Rational regarding CharacterSetValidation and Code Pages

CharacterSetValidation Package is not that good

We have a remaining issue where a user will attempt to update a defect and receive an error from the CharacterSetValidation package. I re-engineered CheckCodePage.pl to check through the Controller database but it didn't find anything. Investigated the CharacterSetValidation package a little more in depth - how does it validate characters? How does it do it differently than me?

Turns out that it merely does:

  if ($line =~ /[^\t\n\r -\177]/) {
    error
  }

This odd regex seems to be merely checking to see if the characters are printable or not. I don't see how this would, for example, prevent a non US ASCII character like é from slipping through!

Narrowing it down a bit with my CheckCodePage.pl (which merely checks that characters are in the range of ordinal 0 - 127 (AKA the US ASCII character set) I figured out that the above regex was failing on characters such as ordinal 7 (Bell) and others. IOW characters that are technically in the US ASCII character set but that are not printable. Not sure how to resolve this problem yet...

Submitted defect to IBM/Rational regarding CharacterSetValidation and CodePages

Recently we upgraded our Clearquest database using 2003.06.15. With this version of Clearquest comes the usage of Code Pages. We choose to set our Code Page to US-ASCII. In upgrading our database we checked to ensure that all character data was within the US-ASCII character set (characters in the range of 0-127). Additionally we installed the CharacterSetValidation package as the Clearquest Administration Guide says:

The CharacterSetValidation package prevents clients running earlier versions of ClearQuest from entering data in a user record from a code page other than the data code page value of that database. If you do not apply the CharacterSetValidation package to your schemas, it is possible for users to enter unsupported data from the client and for data to be corrupted when modified on certain clients.

However it doesn't appear that the CharacterSetValidation package properly prevents unsupported data from being entered. Additionally it actually is causing us problems because characters that are valid US-ASCII are being flagged by the CharacterSetValidation package as unsupported.

US-ASCII is, according to http://en.wikipedia.org/wiki/ASCII:

ASCII is, strictly, a seven-bit code, meaning that it uses the bit patterns representable with seven binary digits (a range of 0 to 127 decimal) to represent character information.

Ergo US-ASCII is the characters whose ordinal values lie in the range of 0-127.

The CharacterSetValidation package installs a few Perl subroutines into the CQ schema that are supposed to check that the character data matches the code page of the database. However there doesn't seem to be any Perl code to check the difference between say a code page of US-ASCII or say Latin-1.

What it all comes down to eventual, as far as I can see, is a call to check_for_control_chars_in_line. The operative Perl code is:

if ($line =~ /[^\t\n\r -\177]/) {
  return (" contains unsupported character(s):");
}

The above regex seems to be saying if the string $line is not a tab (\t), a linefeed (\n), a carriage return (\r) or in the range of space through \177 then an unsupported character has occurred. This does not include the entire US-ASCII range of 0-127. So if a client generated a yen character (¥) the above would not flag it as an unsupported character and yet (¥ - ord 165) is not a US-ASCII character.

Finally, since we only screened for US-ASCII in the 0-127 range, we have data that contains things like the US-ASCII Bel (ord 7) and the like that CharacterSetValidation is flagging as invalid.

Posted by at 4:58 PM | Permalink