Status for Andrew DeFaria

• Spent time trying to understand the Clearcase scheduler and why rgy_backup is not being performed. Apparently modifications were made to the Clearcase scheduler on rtnlprod01 in particular by Paul Kruchmer (or whoever). The net result was the cascading jobs were failing. This should be working more correctly on the new server
• Dave & Busters - Enuff said
• Back at the farm for the server swap: Things went well in the beginning - didn't even need to run reregister script. Then some problems occurred. Eventually we nailed it down to only one real problem - the inability of plain users to create views. Mentioned we should create one plain user in each production group so we can "play" plain user. Eventually Mike discovered that Windows 2003 Server tighened up security on the Share Permissions (not the permissions of the shared folder but the permissions of the share itself) such that Full Control for Everybody is now by default off. Toggled that on and all is well
• Added lockvobs jobs to new rtnlprod01. Script was failing due to recent relative path fixes. Works from the shell but not from the Clearcase Scheduler perspective. Backed out changes for now

Posted by at 11:14 AM | Permalink

• Attended 3 meetings today which, by far, occupied most of my time
• Script review meeting was good. In particular I've implemented the following changes to reregister
• Script not loops through regions for vobs as well as views
• Script now uses -host on lsvob and lsregion to restrict execution to the vob and view hosts only
• Script mktag's for both public and private vobs
• Cleaned up the registry a little bit. Subba created some vobs in cs-nt that were:
  1. Not public
  2. Not in the pmo-nt region
  
  Also cleaned up a vob or two that was tagged in another region as private while it was public in the pmo-nt region. This happens semi-silently in that if you mktag -vob -region <other region>... the mktag will prompt for registry password, giving you the illusion that it is making the tag in other region as public when it is not. If you are mktag'ing into another region a public vob you must explicitly state -public!
• Investigated backup registry service. It is not working in the current environment. Here's how I believe it is supposed to work:
  • You must, of course, configure backup registry server setting on all clients to point to the backup registry server properly. This we had already done.
  • You can manually take a snapshot of the registry by executing rgy_backup. If run on any machine other than the designated backup registry server it will merely ask the current registry server what the backup registry server is and set the backup registry server key in the Windows registry. If run on the backup registry server it will copy a snapshot of the registry files in $CCASE_DIR/var/rgy/backup. It appends a date timestamp to the file name and in the non timestamped filename, records a pointer to this file (e.g. the backup/vob_tag file contains the machine relative path to vob_tag_).
  • Clearcase relies on the Clearcase scheduler and the Standard Daily Job to perform daily snapshots of the registry via rgy_backup. There are two problems with this:
    1. Currently the Standard Daily Job is not even scheduled by default!
    2. One can envision crude developing in the backup directory after months of these daily jobs! I'm not sure how or if there is a mechanism for cleaning up such stuff

Posted by at 7:04 PM | Permalink

• Sent email to Rory Valle for him to install the backup software and start backing up the new servers
• Already tested the transfer of [un]lock vobs jobs. Need to have the new servers up and to activate this to test them
• Scheduled review meeting for reregister script
• Checked the local users/groups on the old and new servers and they appear to be OK
• Re: Genius move: Verified that EntData_ARCH wants R/W access for others denied but Core_automation will go with the standard vob permissions

Posted by at 5:17 PM | Permalink

• Recieved laptop today
• Fixed problem with tables for IE
• Worked on CRPS
• Figured out how to move Genius vobs
• Helped Subba with a vob import

Laptop and Password Issues

When getting my laptop I was asked to change my password so the person could login as me and set up my laptop. I did this. After completing the laptop setup I attempted to change my password back to what it was before. This failed. I then tried to change my password to something else. This also failed. I called Helpdesk and they set my password to Summer1 and told me to log out and back in and I should be prompted to set my password. I did this but was not prompted to set my password. So I hit Control-Alt-Del and selected Change password. I then specified Summer1 as my existing password and selected yet another new, unique password (conforming to the Password Policies, of course). I received the message "The password cannot be changed at this time". Called HelpDesk and they told me that I must wait a while and then I could change my password. Waited a day. Reattempted to change my password. Same message.

Meantime the Clearcase License Monitoring, which is running on my desktop, and the web pages also on my desktop were unavailable while I was rebooting my system and working on this problem.

For your reference I took this opportunity to look up exactly what "The password cannot be changed at this time" according to Microsoft. The article is here. According to it you need to make a change to the Minimum Password Age in Active Directory to change it from Not Defined to something defined (like 0 or perhaps something else).

Genius VOB move

As you know we had difficulties performing the move of EntDataARCH and Core_Automation from the Genius region to the standard environment. This was partly due to having the vobadm password changed on us but it was also partly due to the fact that these vobs were created in a "different" environment, thus leaving us with ownership and permissions problems.

I believe I have a fix. Here's what we need to do.

1. Set up another time where we can perform the move
2. Lock the vobs
3. Remove tag for these vobs in the Genius region
4. Re-move (that's re then move not remove! ) the vobs to rtnlprod01
5. Fix permissions¹
6. Register and re-tag these vobs
7. Unlock vobs
8. Configure clients²
9. Test access

Notes:

1. The permissions fix entails running fix_prot -force -root -recurse -chown vobadm -chgrp .vbs, then register and mktag followed by a cleartool protect -chown vobadm -chgrp -recurse . The first fix_prot fixes the protection on various objects but the second cleartool protect is needed to set the owners correctly.
2. Configuration of the clients should be more than just setting their region, CLEARCASE_PRIMARY_GROUP and clearcase_albd service login properly. It should be a complete uninstall of Clearcase followed by a standard Clearcase installation so that we have a proper and known environment.

Posted by at 10:30 PM | Permalink

• Attempted to move Core_automation and EntData_ARCH vobs to prod01 but failed
• Worked with security people to get vobadm password back
• Worked a little to configure RWP for Perl CGI

Vob move problems

First problem with the move of these vobs was that vobadm's password had changed. Therefore I had to perform this vob move in an unfamilar environment. Also, ownerships and permissions on these vobs are odd as they were done before we started standardizing things. As ccadmin1 I was eventually able to copy the vob storage but was not able to register these vobs at the new location. Attempts to do so would fail with

Even after changing ownerships this was still failing. I suggest we take soem time to get the ownerships and permissions standardized on these vobs in the Genuis region before we attempt to move them again (that and perhaps doing some test moving...)

Vobadm password problems

AFAICT somebody changed the vobadm password. Now I'm not pointing any fingers however nobody in our group did and the only other group who knows the vobadm password is... Well you fill in the rest! Anyways, for some reason at about 3 Pm that somebody must have changed the password back.

Posted by at 3:11 PM | Permalink

• Removed web views from rtnlprod03
• Worked with Timmie regarding move of Genius vobs
• Fixed reregister to handle web views
• Worked with Paul to create CM_Docs vob
• Helped Subba import some code and setup stuff for cs-nt
• Started Clearcase Problem Reporting System

Posted by at 2:31 PM | Permalink

• Requested LDAP access from James Portugal
• Spoke to Tom regarding Niku. Says we'll only need to report time and that he'll hold a brown bag seminar for it
• Requested that rtnlprod01/02 be renamed to 04/05
• Converted web pages to new standard template
• Set up RWP on prodfix01 so that it can access user FAQs
• Checked that Clearcase web is working on prodfix02
• Changed reregister to handle UCM PVOBS in a more intelligent way

Posted by at 7:56 PM | Permalink

• M drive problem appears to be a networking problem at Eric's location
• Condensed Client release areas to one release area
• Attempted to access LDAP through Perl

M drive problem appears to be a networking problem at Eric's location

Today Eric sent over somebody to install and setup Weblogic on my desktop in an attempt to reproduce the problem with Jbuilder writing to the M drive. After much installation, setup and configuration the problem was not reproducable. IOW it worked like a champ from my desktop.

During the installation we copied over the VIP sources instead of using the sources in the vob and we got these from Eric's desktop. It took a very long time leading me to further believe that there are networking problems either on Eric's desktop or somewhere between his desktop and the servers.

Condensed Client release areas to one release area

On //rtnlprod03/CC_REL_2003.6 there exists the following client release areas:

• Training-client
• asap-tavant-client
• resolve-client
• asap-integration-client
• eag-client
• scc-client
• asap-qa-client
• pmo-client
• vip-client

and a server area:

• pmo-server

TUP exists at //rtnlprod03/sqldb/TUP and XDE-Java at //rtnlprod03/sqldb/XDE-Java.

The Clearcase client and server installation areas have all been condensed to one Server and one Client installation area (//rtnlprod03/CC_REL_2003.6/Server and Client). Instead we now have .reg file that set the uniquely different configuration settings for the various groups in the CM_TOOLS vob under etc with names like ASAP.reg, SCC.reg, EAG-ESB.reg. After a client install we will send a link to these files that will cause the settings to be installed on their desktop.

Currently the various client reg files have only the setting of Region, RegBackup and CLEARCASE_PRIMARY_GROUP.

Attempted to access LDAP through Perl

I installed Perl modules to deal with LDAP in the hopes of being able to set up a web page that will list out the membership of the various CC groups from Active Directory. Trouble is I do not know the server name of Active Directory nor the distinguished name to use in order to access this data.

Posted by at 4:29 PM | Permalink

• Finished reregister script. This script will reregister all vobs and views that were based on rtnlprod02. This is needed since the new server will not have the same IP address as the old server.
• Changed mktriggers to use ccperl unqualified instead of fully qualifying it. This will allow the triggers to work on the servers as well as the clients
• Added FAQ entry describing view private and derived objects
• Fixed the FAQ pages. Mike had switched rtnlprod02 over to using rwp. This broke the FAQ pages since they used Apache under Cygwin. Reported rwp to the old FAQ pages.

Posted by at 4:21 PM | Permalink

• Worked with Mike trying to get Clearcase web to function correctly
• Since the IP address of the new servers will be different I worked on a script that will reregister views and vobs. This reregistering and retragging should fix up the IP address problem

Posted by at 10:34 AM | Permalink

• Partitioned drive

• Installed Windows XP

• Attended Peregrine Training

• Configured Clearquest web

• Worked with Michael on Clearcase web problems

• Worked on problem with accessing Clearcase due to outgrowing the lockmgr's limits

• Implemented Heap Size fix and Lockmgr fix to new production servers

Several people have been reporting odd problems with Clearcase. In investigating this I checked the server log files and found a lot of:

*** db_VISTA database error -922 - the lock manager is busy

Searching Rational's web I saw lock manager problems, settings and guidelines which seems to indicate that we've probably already outgrown the default parameters for the lockmgr. The article talks about monitoring the server processes for a week to determine how to best set the values for lockmgr. Unfortunately we don't have a week to wait so for the mean time I simply set the following parms on the lockmgr and restarted Clearcase on the vob sever:

-a almd -u 100 -f 200 -q 500

Posted by at 7:37 PM | Permalink

The new servers have arrived and I have been testing them today. Here's what I managed to do:

• Installed Clearcase server software on new machines.
• Configured storage locations for vobs/views
• Practiced moving a vob and view over to the new servers. The view even had a checked out file.
• Once set up on the new servers accessed the view and insured that I could access it and that the checked out element was still accessible.

In setting up prodfix01 as a the vob server I encountered some problems attempting to set the storage location. The default wizard was not the right way to go. Instead I had to:

Naturally this will not due as this machine will eventually be renamed as rtnlprod01, not prodfix01. This will need to be adjusted during the downtime. Just some notes from my testing session:

After setting up the servers I had to create the storage locations as above. I called them vobstore and viewstore instead of ccvobstg/ccviewstg. Personally I find the former more descriptive.

I tried to simply mktag for the vob and view on the new server but that would not work. Instead I had to:

cleartool register -replace -view \\path\to\view
cleartool register -replace -vob \\path\to\vob
cleartool mktag -replace -view -tag \\path\to\view
cleartool mktag -replace -vob -tag \\path\to\vob
cleartool startview
cleartool mount

Again, if we slide the server into place after changing the DNS/Machine name <-> IP address mapping this should not be an issue.

We should be ready to go, perhaps this coming Thursday evening (sometime after 7 Pm). What we would need to do is:

1. Announce to the Clearcase community that Clearcase will be down for a few hours Thursday night (7 Pm - 9 Pm)
2. At 7 Pm stop Clearcase on rtnlprod01 and rtnlprod02
3. xcopy /e /i /f /h /k /x > C:\vobstore_xcopy.log 2>&1
4. xcopy /e /i /f /h /k /x > C:\viewstore_xcopy.log 2>&1
5. Shutdown rtnlprod01 and rtnlprod02
6. Switch DNS so that prodfix01 -> rtnlprod01 and prodfix02 -> rtnlprod02
7. Test vob/view access from desktop clients.

Posted by at 6:11 PM | Permalink

Imported source into many vobs today with Subba. Had some problems with the vob server (db_VISTA error -922). Ended up stopping and restarting Clearcase on rtnlprod01.

Was assigned to set up both Clearcase web and Clearquest web. Luckily Clearcase web is already up on rtnlprod03. Need to set up Clearquest web...

Modified protect trigger to use current owner instead of original creator of the VOB to change the ownership of new elements.

Posted by at 1:59 PM | Permalink

Re-wrote the lock vobs script in Perl to be much more flexible. It now locks or unlocks (-u) vobs. Also you can supply an -smtphost which defaults to notesmail01. Finally you can specify either -to or -errors-to as a list of email addresses that the report is send to. The later, -errors-to, are for people who only want to be notified if there is an error (i.e. me! :-).

Changed bat files accordingly and tested out adding these to the Clearcase Job Scheduler on my machine. Have not yet added this to "production" (rtnlprod01). Perhaps tomorrow...

Also worked with Subba to clean up and clear out some UCM vobs.

Spend some time trying to get MySQL running under Cygwin.

Posted by at 6:54 PM | Permalink

Came in today and noticed that Notes was complaining that a network operation had failed. Also saw a couple of messages that Clearcase was having problems so I started investigating. Apparently the albd_server was not running on rtnlprod01. Scanning the logs I found:

albd_log: Error: Unable to connect to SMTP server "172.16.101.56[6400]": Bad file descriptor.

Well 172.16.101.56 happens to be notesadmin01:

C09-272-A:nslookup 172.16.101.56
Server:  dhcp01.ameriquest.net
Address:  172.16.101.100

Name:    notesadmin01.ameriquest.net
Address:  172.16.101.56

Why are we trying to email through notesadmin01 instead of notesmail01?

Meantime I unlocked the vobs...

Looking at the SMTP setting in Control Panel: Clearcase: Options I find the SMTP Host set to appsmtp.ameriquest.net which is an alias to notestadmin01.ameriquest.net:

C09-272-A:nslookup appsmtp
Server:  dhcp01.ameriquest.net
Address:  172.16.101.100

Name:    notesadmin01.ameriquest.net
Address:  172.16.101.56
Aliases:  appsmtp.ameriquest.net

I guess the questions are:

• What is the official SMTP host that we can rely on?
• Why does the [un]lock vobs job have problems and eventually kill the albd_server process simply because it cannot contact the SMTP host?

Further investigation yields the following: The lock and unlock vobs scripts apparently appear in .../Rational/Clearcase/var/scheduler/tasks. There are 4 files involved:

• ccase_lock_vobs.bat: Simple bat file that fires off Perl on...
• lock_vobs.pl: This locks the vobs and sends email
• ccase_unlock_vobs.bat: Simple bat file that fires off Perl on...
• unlock_vobs.pl: This unlocks the vob and sends email

Additionally these Perl scripts use C:/Winnt/System32/blat.exe to send mail.

Finally the task_registry file was modified to add these custom jobs.

I believe that this was done by perhaps Paul and/or Brian and it is fine work and does the job. However, seeing as we are about to reburn this system to Windows 2003 Server such work would be lost! And who would remember where this blat came from and that it needed to be reinstalled?

Suggestions:

I think we should use the Rational supplied tool, notify, instead of blat. This way we would not need to remember to find and reinstall this blat thing. I think Brian had problems getting notify to work and instead fell back on something he knew, blat, to get this working. Additionally IMHO blat should not be in the Windows directory! If we must use blat then perhaps we should install it into CM_TOOLS/bin.

Additionally I suggest that we relocate the .bat files and Perl scripts to CM_TOOLS/bin also and task_registry file should be pointed to CM_TOOLS/bin.

Finally I think there should be a script set up to reproduce/reinstall this environment.

IOW our stuff should also be version controlled and scripts written to automate it's installation and workings.

Posted by at 6:35 PM | Permalink