Status for Andrew DeFaria

• Changed Dynamic Lists to properly handle bad data

Dynamic Lists and Bad Data

I now have pqamerge properly creating Dynamic Lists. In the past I would call SetFieldChoiceList, which would allow validate and commit to work, but which wouldn't actually add a member to the Dynamic List. I've since changed that.

The first problem I hit was that Clearquest would treat "1.A" as the same as "1.a". So when pqamerge attempted to add "1.a" to the Dynamic List it would fail. I needed to compare ignoring case. In Perl usually one uses a regex for that and specifies "i" to ignore case.

if ($var =~ /^$pattern$/i) {
  <matched ignoring case>
} # if

But some of the values of the various Dynamic Lists contain regex meta characters (e.g. "?" or "(") which caused me problems. I've fixed those now.

if ("\L$var\E" eq "\L$pattern\E") {
  <matched ignoring case>
} # if

The above basically says to downshift characters between the "\L" and the "\E". Then a regular eq is used so as to avoid problems with interpreting regex meta characters.

Posted by at 1:50 PM | Permalink

• Fixed bug in pqamerge so that it now properly adds new entries to dynamic lists

AddListMember

I was dynamically adding to Dynamic lists by calling SetFieldChoiceList. This allowed validate and commit to work and for the record to be added however it doesn't actually add to the Dynamic list! You can see this later by attempting to modify a record. It's dynamic list items will show in red and you will not be able to apply the changes because it will state that the value is not a valid choice of that dynamic list.

This is quite a surprise to me. I found that I need to call AddListMember to actually get this to add to the Dynamic list. This appears to be working so I need to redo the merge again.

Posted by at 1:51 PM | Permalink

• Wrote and enable_ldap Perl script
• Vinh Ton has provided me with yet another schema and corresponding databases

Another Merge

Performing yet another merge so I will once again document the steps I take:

• Restored the databases CQSchema03, CQ_Controller_Test and CQ_Controller_Prod. Doing so wipes out the old data
• When importing you must go into the CQSchema03 tables for master_dbs and change the server to p4test. Otherwise you will be effecting the data on the other server!
• Enter Clearquest Designer and mark the following fields as optional:
• Submitter
• Submit_Date
• Audit_Log
• old_id
• Change commitmentLevel_Value_Changed hook to simply return
• Test Work, Check in Schema and Upgrade Database
• Import OEMUsers.cqu and Users.cqu
• Upgrade Database
• Run pqamerge
• Test
• Reverse the schema changes performed above and apply the changes
• Backup new CQSchema03, CQ_Controller_Test and CQ_Controller_Prod

Use the following for a Check in Comment:

Changing Submitter, Submit_Date, Audit_Log and old_id to be optional for data conversion

Also changed commitmentLevel_ValueChanged hook to return before sending email.

Posted by at 6:19 PM | Permalink

• Resolved most of the remaining issues with PQA Merge Vinh Ton

Remaining Issues

I have incorporated all of your changes and have imported your CQ_Controller_Prod, CQ_Controller_Test and CQSchema03 databases. I don't see any new users. Weren't they in the CQAllUsers database?

In any event here's how I proceeded:

• Made changes to pqamerge. This included handling a few new fields that I didn't notice before as well as adjustments to accommodate the changes in your last email. Also set Cont:old_id = the old id from the TO or Prod. Now we have a cross reference field. You will need to change your form to expose this field to the users.
• Imported CQ_Controller_Prod, CQ_Controller_Test and CQSchema03. I see the changes that you've made to the schema.
• Imported OEMUsers.cqu and Users.cqu.
• Changed Submitter, Submit_Date, Audit_Log, and old_id to be optional
• Changed Global Script: RecordHistory to simply Exit Function
• Changed CommitmentLevel_ValueChanged to Exit Sub before sending email.
• Added delete action to defect record for admin only. This is needed for pqaclean to work.
• Ran pqamerge

By and large this worked! There was a user who was missing. This user, tngo, was not listed as being subscribed to either TO nor Prod, rather to BT. I suspect that he used to be subscribed to Prod but then later was removed. If there are more users like this then there will be similar errors.

Posted by at 5:23 PM | Permalink

• Starting to look into user accounts issues
• Send out form for LDAP Authentication Information
• Coded up pong.sh and pong.tcl to check and see if ccase-rmna-3 is up or down
• Created initial set of PQA User/Group Account files

PQA User/Group Account Files

Here's what I did for user accounts. I've attached all the files and will refer to them by name here.

OEM Users

First I took the export list from Clearquest (userinfo.txt.orig). This is the untouched original export file.

Next I extracted user names that begin with "oem" -> OEMUsers.cqu. Then I noticed that some of these users only appeared to be subscribed to NAS or BT. I then separated those out to OldOEMUsers.cqu.

New Groups

Patterning off of userinfo.txt.orig file for groups I created NewGroups.cqu to match the new groups you describe on your Users tab of the Excel Workbook. The group records from userinfo.txt.orig I stored off in Groups.cqu and then separated off groups that were only NAS or BT to OldGroups.cqu.

Other Users

This left me with all of the other users which were split up into Users.cqu and RemovedUsers.cqu based on whether they subscribed to Prod or TO.

Changing databases= line

For all files I made the following assumption: If the databases= line contained no values then that means "all databases". So any users or groups with a databases= line with no values I assume subscribed to TO or Prod. I left these databases= lines alone.

For other users or groups, if TO or Prod appeared in the databases= line I removed all values and put in Cont.

Please let me know if this comes close to what you wanted WRT users and groups.

Posted by at 3:22 PM | Permalink

• Documented remaining PQA Conversion Issues
• Audit_Log is not handled properly currently. Querying Vinh
• Resolved problem with Submitter and Submit_Date. Need to change schema to make those fields OPTIONAL as well as add return statements to the Perl action hooks
• Investigated using LDAP authentication with Clearquest
• Wrote a simple expect script (pong) and bash script to verify that a machine is up and send email if it isn't. Problem is ccase-rmna-1 lacks expect!

Audit_Log

As it stands Audit_Log is probably not being handled the way you think it should be. My investigation seems to indicate that Audit_Log is, as its name suggest, an auditing of what changes were made. In transferring from the old databases to the new one, the new Audit_Log is essentially a blank slate. It is the responsbility of the Action Script RecordHistory to update the Audit_Log. Thus the only recording in Audit_Log for the transfer is that the record was added! In other words all history is lost! I don't think that's what you want. We will need to similarly change Audit_Log to be a simple OPTIONAL field with no Action Scripts in order for pqamerge to transfer the Audit_Log.

LDAP Authentication

Regarding LDAP authentication: There is a whole new chapter in the Clearquest Admin Guide about this and we'll need to coordinate this with an LDAP Administrator (Ray, who's our LDAP Administrator?). On the plus side users would no longer have to remember separate usernames/passwords for Clearquest. In fact, new users would not even need to be added because if they are in the ActiveDirectory then they have a login from Clearquest. Of course I suspect that security is then achieved by Clearquest groupings, which, as far as I can tell, are still manually created in Clearquest Designer's User Administration screen. Additionally, LDAP permitting, other fields will be instantaneously mapped properly like Full Name, Phone Number and the ever popular Email Address! In other words, LDAP authentication is doable but a little complicated, requires coordination with the LDAP administrator and requires a number of carefully executed commands.

One issue or thing to test would be if older Clearquest clients can still authenticate with LDAP. If not then the user community would need to update their Clearquest.

Pong

I now have a script called pong.sh (and a corresponding expect script called pong.tcl) that will "pong" a system and send email if it does not respond. By "pong" I mean it will telnet to the machine, attempt to login as vobadm and check for a "proper" prompt. If it fails then email is sent to the members of our project (Ray is working on a bona fide email alias...).

I planned on running this from ccase-rmna-1 in vobadm's crontab (BTW: There was a blank line in the crontab, which causes Solaris to complain (Other OSes are this picky. I removed that blank line).

The problem now is that ccase-rmna-1 lacks expect! :-(

Posted by at 8:00 PM | Permalink

• Fixed minor problems with trailing spaces on fields
• Fixed minor problem with space filled Software_Revision
• Fixed problem where field names were reversed.
• Submitter and Submit_Date are set by action hooks to the current username and current date/time. We'll need to turn this off if we want the old Submitter and Submit_Date.
• Implemented Attachment Handling

Trailing spaces

There are some minor problems with the data involving spaces. For example, a Prod:HUT_Version is set to "BCM95704CA40 v1.0 revA0 " - Note that it has a trailing space. Well AddToFieldChoiceList() attempts to add this value but the Clearquest API trims the trailing space! This results in a validation error as "BCM95704CA40 v1.0 revA0 " is not equal to "BCM95704CA40 v1.0 revA0". The solution here is to trim the trailing space before calling AddToFieldChoiceList().

Space filled Software_Revision

Prod:Software_Version would sometimes come in as " " instead of "". My code was only checking for "" and changing that to "N/A". It now also checks for " ".

Field name reversal

Some fields, for example ReportedBy, were supposed to be renamed to another field name, i.e. Reported_By, in the destination database but my code reversed the field naming change. It's odd that Clearquest was OK with adding ReportedBy field to Cont as there was no ReportedBy field in Cont. Adding the correct field name solved a problem with Visibility and its action hook.

Attachments take time!

While the TransferAttachments() routine now works, it adds a significant amount of time to the processing of defects with attachments. Some attachments are as large as 82 Meg! There are zip files, Excel spreadsheets, screen dumps in the very inefficient BMP file format and Ethreal dumps and even tape I/O dumps. TransferAttachments() operates by extracting the files from the source database to disk then calling AddAttachment() to copy the files back into the new DB. While not the most efficient way to do this I see no other way given the current Clearquest API.

As a measure, previously it took 16 minutes and 44 seconds to merge 1382 defect records from TO -> Cont, with attachments it took 4 hours and 11 minutes!!!

Ah, figured out why it took so long. I was using a snapshot view that was housed on my laptop. So the attachment files were being written to the current working directory, my snapshot view on the laptop. So those writes and reads were over the network. To make matters worse my laptop was not even at Broadcom, rather it was at my house. So those 82 meg files had to travel from Broadcom though my small DSL connection and even through a VPN! No wonder it took so long!

Now running it in ~vobadm/My Documents on the server and it's much quicker... Only 25 minutes and 21 seconds.

Posted by at 12:57 PM | Permalink