View server had a problem again. More and more it seems to be a problem with the NIC card on rtnlprod02. Managed to log into rtnlprod02 and look around. Couldn't see any processes at all in the TaskManager even though the 4 CPUs were 70-90% busy! Couldn't look at the Network tab - said there was no network adapter. Additionally the following appeared in the System event log:
Event Type: Error Event Source: NIC Agents Event Category: Service Event ID: 1285 Date: 9/21/2004 Time: 2:07:32 PM User: N/A Computer: RTNLPROD02 Description: NIC Agent: Connectivity has been lost for the NIC in slot 0, port 1. [SNMP TRAP: 18006 in CPQNIC.MIB] Data: 0000: 00 00 06 00 00 00 00 00 ........ 0008: 01 00 ..
Held code review for cmconfig.vbs. Spent the rest of the day implementing changes. Some changes were minor while a few others were good catches of possible problems.
Working mainly on cmconfig.vbs. Solution for the env problem was to set both the user environment and the process environment then call cmverfiy.cmd from cmconfig.vbs. The code now does the folowing:
There is a slight problem with the "set everything and verify it" script cmconfig.cmd. The idea was that that one script would perform 3 separate actions:
The problem is with #2 being VBScript and #3 happening as a result of executing from the cmconfig.cmd script. While setccconf.vbs does indeed set the proper configuration environment variables as soon as it ends those settings are no longer set anymore by the evoking script only. All other (new) processes know about the change to the environment variables. To illustrate this odd phenomena:
$ type setenv_var.vbs
Set sh = WScript.CreateObject ("WScript.Shell")
Set env = sh.Environment ("USER")
env ("foo") = "bar"
$ type senenv_var.cmd
@echo off
set foo=bar
$ echo foo = %foo%
foo = %foo%
$ cscript setenv_var.vbs
$ echo foo = %foo%
foo = %foo%
$ call setenv_var.cmd
$ echo foo = %foo%
foo = bar
$ cmd /c echo foo = %foo%
foo = bar
Why it works for a .cmd script but not for a .vbs script I don't know.
So I will change cmconfig.cmd to do #1 and #2 above but skip #3. This means two things:
The reason for #2 is that even after cmconfig.cmd is run environment variables such as CLEARCASE_PRIMARY_GROUP and CLEARCASE_GROUPS will not have been "exported" into that cmd's environment (unless somebody can tell me how to set the parent's environment variables from VBScript...).
For the point and click crowd this should be OK. The email message you send the customer should have two links: one to cmconfig.cmd and one to cmverify.cmd. If the customer clicks the cmconfig.cmd link then cmconfig.cmd should run, in it's own process, and go away. Subsequently if the user clicks on cmverify.cmd that will run in it's own cmd process and it should pick up the appropriate environment variable settings.
There have been many requests for enhancements to the configuration scripts including updating the TOOLS database on successful verification and performing additional checks such as "is this user really in the AD group". I believe I will only be able to solve the above problem by having the setccconf.vbs script actually envoke the cmverify.cmd directly so that the environment variables that setccconf.vbs just set are exported into the cmverify.cmd's environment.
Found that there are situations where one needs to have CLEARCASE_PRIMARY_GROUP set. Worked with Mike to implement groups in the TOOLS database. Spent a large part of today implementing setccconfig.vbs. This command will replace setregion and sets 3 things:
In order to do this the username is used to retrieve information from the TOOLS database. If the username is found then we get the project record and obtain the region. Next we check CC Primary Group and CC Groups. These fields will not always be filled in (previous entries lacked them but future entries should have them filled in. We then set CLEARCASE_PRIMARY_GROUP and CLEARCASE_GROUPS if there are any.
CC Groups in the TOOLS database will contain all Clearcase groups that the user belongs to - including their Primary group. In a lot of cases the user's CC Groups will be only one and equal to the Primary group. In that case CLEARCASE_GROUPS is not set. If the user is a member of more than one group then it will be their Primary group and additional groups in the CC Groups field. In that case Primary group is removed from that list and the rest are set into CLEARCASE_GROUPS.
Worked on setccconf.vbs
Here are the requirements as I understand them for the trigger that Steve Lipson wants for the SQL checkins. Basically he desires a trigger that will capture the checkin comment and other information and insert that information in the form of a comment at the top of the checked in element. This trigger will:
Roughly the psuedo code for this trigger will be:
# Get name of element and its type
$pname = $ENV{CLEARCASE_PN};
$element_type = $ENV{CLEARCASE_ELTYPE};
# Skip directories and elements that aren't .sql
exit if $element_type =~ /directory/i || $pname !~ /\.sql$/i;
# Get comment and user
$comment = $ENV{CLEARCASE_COMMENT};
$userid = $ENV{CLEARCASE_USER};
# Format timestamp
$timestamp = getCurrentTime;
# Parse output of lsactivity -cact -long
($activity_id, $activity_title, $activity_owner) = parseLSActivity;
# Open up $pname for reading and $pname.trig for writting
open PNAME_IN, $pname
or die "Unable to open $pname for reading - $!\n";
open PNAME_OUT, ">$pname.trig"
or dir "Unable to open $pname.trig for writing - $!\n";
# Add comment to top of file
print $PNAME_OUT <<END;
-- Date: $timestamp
-- Activity: $activity_id: $activity_title
-- Owner: $activity_owner ($userid)
-- Comment: $comment
END
# Append $pname
while () {
print PNAME_OUT $_;
} # while
close PNAME;
close PNAME_OUT;
# Switch $pname.trig -> $pname
unlink $pname;
rename "$pname.trig", $pname;
# Allow checkin to proceed
exit 0;
OK, licenses are all OK now. BTW hostid is not a combination of IP address and MAC address - it's simply MAC address of the NIC card. Problem appears to be that during the server move the NIC card changed! I verified this with the Rational rep this morning. The current MAC address of the NIC card as shown by ipconfig /all (it's the Physical Address) is 00-0f-20-6d-09-97. I gave the old license strings to the rep and she determined our old MAC address of the NIC card was 00-0b-cd-9b-f4-27.
Looking at the Device Manager on the servers I notice that we have two Network Adapters listed:
Why we have 2 NIC cards I don't know (perhaps merely for the ability to be multihomed). Normally I see #1 being disabled and #2 being enabled. Rtnlprod01 is like this. Rtnlprod02 has #1 enabled and #2 disabled. For the vob and view servers this does not matter. Either card will work. For rtnlprod03 however, since licenses are tied to the NIC card's MAC address, switching from #1 -> #2 (or #2 -> #1 as I suspect has happened here) will invalidate all licenses. We need to insure that the network adapter for rtnlprod03 remains nailed at either #1 or #2.
Since it appears that the network adapter has changed and since I have obtained a new set of licenses for this new network adapter and still have the old set, I have installed both sets so that in the event that the network adapter switches again we'll be covered. However, if we order additional licenses for say card #1 and somehow we get switched to card #2, those additional licenses will not be represented, hence the desire to nail the NIC card to either one or the other.
Implemented Permissions trigger.
I've given the "permissions trigger" some thought and would like to formalize the requirements a bit. The new trigger will have the following characteristics:
Let's see an example of how this will work and how the trigger will respond. Let's assume the following directory structure:
Core_automationEmpower CC-EAG-AS, CC-EAG-ESBFunctions CC-EAG-VIP
Results CC-EAG-VMS
Common
Further let's assume that the permissions element is at the Empower level and contains the groups CC-EAG-AS and CC-EAG-ESB. This says that those two groups (as well as CC-TTE as primary group owners of the vob) have write permission (the ability to checkout) elements from Core_automation/Empower downward. Additionally let's say that we have a permissions element at Empower/Functions that lists CC-EAG-VIP and Empower/Results has a permissions element that lists CC-EAG-VMS. The following can be said:
The pseudo code for the trigger is roughly as follows. Note that the trigger gets fired during checkout of an element only (it is assumed if the user successfully checked out the element then, at the time, he had write permissions and should be allowed to check in the element):
$vob_group_owner = GetGroupOwner (vob)
$current_group = GetCurrentGroup (CLEARCASE_PRIMARY_GROUP as per "creds")
if (permissions element exists in the current folder) {
if (IsAMember (Parse ($permissions_element), $current_group) {
<allow checkout>
} else {
<recurse to check parent folder stopping at vob root>
<disallow checkout>
}
}
Created various views to perform performance testing of PowerBuilder builds.
The backup script is ready for deployment. It was determined to lock all vobs at once, back them all up then unlock the vobs. I also implemented the 7 days of history option. History is stored under d:\vobstore\backup as numbered directories (0-6 for Sun-Sat). The script now manages that area by first moving the current backup under d:\backup -> d:\vobstore\backup\
I was experiencing "Access denied" errors irradically until I found out that there may be .being-deleted files as described here. To solve this I implemented exclusions on the xcopy command (/exclude:d:\backup\exclude.strings) which tells xcopy to exclude things in the "c" directory - i.e. the cleartext pool - which Clearcase can easily recreate without data loss. This also has the effect of reducing the backup copy size in some cases significantly (e.g. \ASAP_Docs goes from 2.1 gig down to 1 gig!).
Here's a rough psuedo code of what the script does:
Obtain a list of all vobs
Lock all vobs
foreach @vobs
if d:\vobstore\backup\$current_day\$vob exists then
remove it
end if
move d:\backup\$vob -> d:\vobstore\backup\$current_day\$vob
copy $vob's storage area -> d:\backup\$vob
Next
Unlock all vobs
Cutting out the copying of cleartext pool has reduced the current backup time from 56 minutes (yes it grew) to 33 minutes.
Now, where are we WRT the SQL backup scripts?....
Spent some time trying to multisite a test vob (\defaria). Realized that we will have a problem in that we will need a machine (server or client) dedicated to handling the multisite region in order to lock vobs and backup things.
Had a meeting WRT multisite approach. It was determined that will we switch to using a script to make a copy of the vobs and back up from there. Started writing script...
Spent most of the day investigating XML and XSL.
The Clearcase web service died today. I don't know why but the following is what I did to get it back up and running.
Server is back up.
Problem analysis:
Not really sure where things went wrong however when I looked at it the Rational Web Platform (rwp - which is really just Apache with Tomcat under the hood) was not running. rwp creates error logs but they are hard to parse because there are many of them with apparently random numbers tacked on. I believe these numbers represent the date in some form but haven't figured out how to decode them.
Anyway, I tried moving them aside and to restart the rwp server so that it would be clear what the error file was. Couldn't move all the log files out of the way because some were busy. Wish I had a good tool to figure out who has a file open.
The error message when trying to start rwp was "The Rational Web Platform HTTP Service Terminated with Service-Specific Error 1". Searching the Rational Support site yielded that the reason the service could not start was because some other process was using port 80. Typically this is due to somebody else runnig IIS. No IIS running. Tried netstat but could not find who was using port 80. Tried telneting to port 80 and got "Connection failed" so whoever was using port 80 did not want to talk to me! :-)
Finally I tried running rwp.exe from the command line with --help hoping for help about rwp's possible command line opts. Instead rwp started and ran! Clearcase web was back up but I did not like having rwp running in a console window and if I were to log out I feared that the rwp process would end. So I went to the Services applet and tried to start (since there was no stop) rwp. It started and the rwp running in the console ended. For safety's sake I restarted the rwp service through the Services applet. I believe Clearcase web is OK for now and we are doign server maintainance tonight anyway...
Worked on getccgroups.vbs most of the day. Managed to get it to recurse for groups within groups and to sort the arrays of names. Need to implement the email portion though.
So it seems that there is some process in the Clearcase Explorer that requires that the share point have Everyone: Full Control set (or at least something more than just CC-PMO and CC-TTE). Note that if I click somewhere out of the directory, like to the subfolder then go back I get the "mvfs: ERROR..." again. I see no errors in the cleartool command line - just in Clearcase Explorer.
Seems the answer to the problem is to add clearcase_albd to the share permissions. I also added vobadm and the CCADMIN group
Spend most of my time setting up a test environment for the web pages. Problem is that the web server (Apache) cannot see the M drive when started as a service so it must refer to a snapshot view. Created webview for this purpose. But how does on test when in order to do so would require to publish your changes?
So I created another view, webtest. But how does Apache reference two different DocumentRoot's? We don't have the power of aliasing, and there's the two web servers to consider: c09-272-a and rtnlprod02.
Ended up configuring Apache to listen on two different ports at the same time using VirtualHosts on both servers.
Then I hit a problem with snapshot views not honoring symlinks properly - it copies the elements instead - and difficulty in updating such things. You see I was attempting to share css and images folders by means of a symlink. Abandoned that idea. Still need to hammer out where I have full URL's and pay attention to the different ports, etc...
Emailed Rory Valle regarding backups, he says they are being done. Also prodded James Portugal again about LDAP access. Finally attended the CM Staff meeting
First problem with the move of these vobs was that vobadm's password had changed. Therefore I had to perform this vob move in an unfamilar environment. Also, ownerships and permissions on these vobs are odd as they were done before we started standardizing things. As ccadmin1 I was eventually able to copy the vob storage but was not able to register these vobs at the new location. Attempts to do so would fail with
Even after changing ownerships this was still failing. I suggest we take soem time to get the ownerships and permissions standardized on these vobs in the Genuis region before we attempt to move them again (that and perhaps doing some test moving...)
AFAICT somebody changed the vobadm password. Now I'm not pointing any fingers however nobody in our group did and the only other group who knows the vobadm password is... Well you fill in the rest! Anyways, for some reason at about 3 Pm that somebody must have changed the password back.
Continue reading "M Drive problem, LDAP stuff, Installation Areas" »
Partitioned drive
Installed Windows XP
Attended Peregrine Training
Configured Clearquest web
Worked with Michael on Clearcase web problems
Worked on problem with accessing Clearcase due to outgrowing the lockmgr's limits
Implemented Heap Size fix and Lockmgr fix to new production servers
The new servers have arrived and I have been testing them today. Here's what I managed to do:
In setting up prodfix01 as a the vob server I encountered some problems attempting to set the storage location. The default wizard was not the right way to go. Instead I had to:
Naturally this will not due as this machine will eventually be renamed as rtnlprod01, not prodfix01. This will need to be adjusted during the downtime. Just some notes from my testing session:
After setting up the servers I had to create the storage locations as above. I called them vobstore and viewstore instead of ccvobstg/ccviewstg. Personally I find the former more descriptive.
I tried to simply mktag for the vob and view on the new server but that would not work. Instead I had to:
cleartool register -replace -view \\path\to\view
cleartool register -replace -vob \\path\to\vob
cleartool mktag -replace -view -tag\\path\to\view
cleartool mktag -replace -vob -tag\\path\to\vob
cleartool startview
cleartool mount
Again, if we slide the server into place after changing the DNS/Machine name <-> IP address mapping this should not be an issue.
We should be ready to go, perhaps this coming Thursday evening (sometime after 7 Pm). What we would need to do is:
Imported source into many vobs today with Subba. Had some problems with the vob server (db_VISTA error -922). Ended up stopping and restarting Clearcase on rtnlprod01.
Was assigned to set up both Clearcase web and Clearquest web. Luckily Clearcase web is already up on rtnlprod03. Need to set up Clearquest web...
Modified protect trigger to use current owner instead of original creator of the VOB to change the ownership of new elements.
Re-wrote the lock vobs script in Perl to be much more flexible. It now locks or unlocks (-u) vobs. Also you can supply an -smtphost which defaults to notesmail01. Finally you can specify either -to or -errors-to as a list of email addresses that the report is send to. The later, -errors-to, are for people who only want to be notified if there is an error (i.e. me! :-).
Changed bat files accordingly and tested out adding these to the Clearcase Job Scheduler on my machine. Have not yet added this to "production" (rtnlprod01). Perhaps tomorrow...
Also worked with Subba to clean up and clear out some UCM vobs.
Spend some time trying to get MySQL running under Cygwin.
Came in today and noticed that Notes was complaining that a network operation had failed. Also saw a couple of messages that Clearcase was having problems so I started investigating. Apparently the albd_server was not running on rtnlprod01. Scanning the logs I found:
albd_log: Error: Unable to connect to SMTP server "172.16.101.56[6400]": Bad file descriptor.
Well 172.16.101.56 happens to be notesadmin01:
C09-272-A:nslookup 172.16.101.56 Server: dhcp01.ameriquest.net Address: 172.16.101.100 Name: notesadmin01.ameriquest.net Address: 172.16.101.56
Why are we trying to email through notesadmin01 instead of notesmail01?
Meantime I unlocked the vobs...
Looking at the SMTP setting in Control Panel: Clearcase: Options I find the SMTP Host set to appsmtp.ameriquest.net which is an alias to notestadmin01.ameriquest.net:
C09-272-A:nslookup appsmtp Server: dhcp01.ameriquest.net Address: 172.16.101.100 Name: notesadmin01.ameriquest.net Address: 172.16.101.56 Aliases: appsmtp.ameriquest.net
I guess the questions are:
Further investigation yields the following: The lock and unlock vobs scripts apparently appear in .../Rational/Clearcase/var/scheduler/tasks. There are 4 files involved:
Additionally these Perl scripts use C:/Winnt/System32/blat.exe to send mail.
Finally the task_registry file was modified to add these custom jobs.
I believe that this was done by perhaps Paul and/or Brian and it is fine work and does the job. However, seeing as we are about to reburn this system to Windows 2003 Server such work would be lost! And who would remember where this blat came from and that it needed to be reinstalled?
I think we should use the Rational supplied tool, notify, instead of blat. This way we would not need to remember to find and reinstall this blat thing. I think Brian had problems getting notify to work and instead fell back on something he knew, blat, to get this working. Additionally IMHO blat should not be in the Windows directory! If we must use blat then perhaps we should install it into CM_TOOLS/bin.
Additionally I suggest that we relocate the .bat files and Perl scripts to CM_TOOLS/bin also and task_registry file should be pointed to CM_TOOLS/bin.
Finally I think there should be a script set up to reproduce/reinstall this environment.
IOW our stuff should also be version controlled and scripts written to automate it's installation and workings.
Ah the comfort of being able to get through the firewall! It feels so good. Now I can read email from my home and use IM again! Yippie!
Alas my system will be moving in two days as I am moving too! Things will be hectic for the short term.
At work I worked on the script to add triggers to the vobs a bit more. Still need to convert it to Perl...
Started investigating adding storage to the servers. This may get a little tricky. Need to meet with Brian and Babu about this.
Played with the web server some more - now pointing it into the actual vob. Had a hell of time doing this as things were acting weird. Rebooted my system and wham! Everythings good now...
This afternoon, after returning from the training I did:
Worked on problem with case perserving MVFS. Two of the servers had case perserving set off so I fixed that. While working on another problem one of those servers rebooted. Oh well, at least that fixes it's case preserving. The other server was later rebooted. Still clients need to make sure that they have case preserving set on and need to reboot. Unfortunately this was not set properly during the rollout.
Worked on a problem view with no success. Unable to get it to work. Keep getting unknown error in view server - not very helpful.
Got my user ID finally! Was able to run Lotus Notes. Oh boy, what joy! Anyways I was able to send email via Ameriquest's email server. Still can't pop nor imap it though.
Installed Cygwin on some of the servers as well as set up my environment. Need to get my new user ID added into the Clearcase group. Can now ssh into some of the servers to perform actions.
Set up Apache under Cygwin on one server. Babu asked me to see if I could make a read only web server to view thing.
Well I got an apartment! It's called The Lake and it sure has a large lake. The apartment is a 1 bedroom with a loft and it's 1000 square feet. I have the lake right off my patio. This means that I need to move in this coming weekend! Gotta get busy.
My car's transmission decided to die so I could not drive down and had to fly in instead. Well the flight was nice but it was too damn short! :-) Looks like I'll need a new car. Wonder if I can get the Eclipse back...
Today at work I get in and started downloading the tarball of my stuff. Wonder when I'll get my own userid? Then Babu tells me to come with him. Off to another Clearcase Training class that he's holding in another Ameriquest building (Apparently there are a lot of Ameriquest buildings and we might be moving to one of them. This other building was brand new and quite nice). The class was OK but they were not ready for us so we had to make due with a small conference room and just use Babu's laptop for class exercises. Then Babu cuts out and leaves me to finish up the training! It's only my 3rd day here and I'm supposed to train!?! Oh well. It wasn't that bad but I was not prepared for that.
Got a ton of things to do and to keep track of
Long day yesterday (9 Am - 11 Pm).
Today I:
Attended the 9 Am meeting
Wasn't able to lock the old ASAP_Docs vob so I rm'ed the tag instead
Checked in the CheckinPre trigger
Setup web server on my deskctop
Removed a number of test vobs
Well made it here. We're on the 9th floor of this building. Argh, having to go through all of the new things like badges, user ids, etc. Apparently this shop is much more closed up than the old place. I cannot, for example, get through to my machine at home for email and the like. I can get to my home page. Guess I'm gonna have to implement that web mail thing after all!
Can't ftp home, can't use IM. No IMAP. No POP. Apparently they also restrict other web sites like sports related stuff, etc.
Interesting in that they apparently use Lotus Notes for email. Argh! This is gonna be a fight regarding tools and the like.
Created a vob, view, etc today. We are supposed to be doing a lot of client installs today and tomorrow.
Managed to install Cygwin partially and get some semblence of my environment.