Migrating colocated views from 6.0 -> 7.0
I performed a test of my snapshot view p6258c_SIMCQ00000392_intview. This view is a colocated view and exists in my home directory (~p6258c). Clearcase administration of the registry can be tricky - you need to get several things correct before it will work.
Here's what the view looks like:
$ ct lsview -l -full -properties p6258c_SIMCQ00000392_intview Tag: p6258c_SIMCQ00000392_intview Global path: /net/cclinux/export/home/p6258c/p6258c_SIMCQ00000392_intview/.view.stg Server host: seast1 Region: cclinux Active: NO View tag uuid:90c8b0d3.d9c311dc.92fe.00:09:3d:14:f4:70 View on host: seast1 View server access path: /home/p6258c/p6258c_SIMCQ00000392_intview/.view.stg View uuid: 90c8b0d3.d9c311dc.92fe.00:09:3d:14:f4:70 View attributes: snapshot,ucmview View owner: rantest/p6258c Created 2008-02-12T16:31:00-07 by p6258c.cclinux@ranlin04 Last modified 2008-03-28T13:53:02-07 by p6258c.cclinux@raneast Last accessed 2008-03-28T13:53:02-07 by p6258c.cclinux@raneast Last read of private data 2008-03-28T13:53:02-07 by p6258c.cclinux@raneast Last config spec update 2008-03-28T13:22:18-07 by p6258c.cclinux@raneast Last view private object update 2008-03-28T13:53:02-07 by p6258c.cclinux@raneastText mode: unix Properties: snapshot ucmview readwrite Owner: rantest/p6258c : rwx (all) Group: rantest/cclinux : r-x (read) Other: : r-x (read) Additional groups: rantest/east
Here's what I did
- cd'ed into this snapshot view ~p6258c/p6258c_SIMCQ00000392_intview/vobs/simdev/bin
- Checked out a script named east.
- cd'ed out of the snapshot view to my home directory.
We now have a colocated snapshot view created on CC 6.0 and hosted by seast1. The storage (my home directory) oddly enough is served by cclinux...
- Moved the view database over to cclinux under it's gdviews
partition:
Note:
Cclinux:mv /home/p6258c/p6258c_SIMCQ00000392_intview/.view.stg \ > /gdviews/p6258c/p6258c_SIMCQ00000392_intview.vws
- In the above command we also renamed .view.stg -> <viewname>.vws.
- We did a mv instead of cp. I suspect that if you are in the snapshot view storage area and you perform a Clearcase operation, cleartool will notice the .view.stg directory at the snapshot view's root and not bother to attempt to contact the registry server to find out where the view database is. Instead it just says "Well the view database is right here". This would allow a disconnected, but colocated snapshot view to be able to perform Clearcase operations while disconnected.
- Reregistered the view:
Note:
Cclinux:ct register -view -replace -host cclinux \ > -hpath /gdviews/p6258c/p6258c_SIMCQ00000392.vws \ > /net/cclinux/gdviews/p6258c/p6258c_SIMCQ00000392.vws
- The -host changed from seast1 -> cclinux
- The -hpath changed to be relative to /gdviews (where we moved the .view.stg)
- Retaged the view:
- Again the -host changed from seast1 -> cclinux
- The -gpath changed to a global path to /gdviews
- Stop the view server:
- Set the config spec to current since the compiled config spec still thinks it's on the old server. Note this will involve a ct update too. Also, we need to cd to the snapshot view storage area:
Cclinux:ct mktag -replace -view -tag p6258c_SIMCQ00000392_intview \ > -host cclinux \ > -gpath /net/cclinux/gdviews/p6258c/p6258c_SIMCQ00000392_intview.vws \ > /net/cclinux/gdviews/p6258c/p6258c_SIMCQ00000392_intview.vws
Cclinux:ct endview -server p6258c_SIMCQ00000392_intview
Cclinux:cd ~/p6258c_SIMCQ00000392_intview Cclinux:ct setcs -current > /dev/null 2>&1Now we have the following view:
$ ct lsview -l -full -properties p6258c_SIMCQ00000392_intview Tag: p6258c_SIMCQ00000392_intview Global path: /net/cclinux/gdviews/p6258c/p6258c_SIMCQ00000392_intview.vws Server host: cclinux Region: cclinux Active: NO View tag uuid:90c8b0d3.d9c311dc.92fe.00:09:3d:14:f4:70 View on host: cclinux View server access path: /gdviews/p6258c/p6258c_SIMCQ00000392_intview.vws View uuid: 90c8b0d3.d9c311dc.92fe.00:09:3d:14:f4:70 View attributes: snapshot,ucmview View owner: rantest/p6258c Created 2008-02-12T16:31:00-07 by p6258c.cclinux@ranlin04 Last modified 2008-03-28T14:32:44-07 by p6258c.cclinux@ranlin04 Last accessed 2008-03-28T14:32:44-07 by p6258c.cclinux@ranlin04 Last read of private data 2008-03-28T14:32:44-07 by p6258c.cclinux@ranlin04 Last config spec update 2008-03-28T14:32:42-07 by p6258c.cclinux@ranlin04 Last view private object update 2008-03-28T14:32:44-07 by p6258c.cclinux@ranlin04 Text mode: unix Properties: snapshot ucmview readwrite Owner: rantest/p6258c : rwx (all) Group: rantest/cclinux : r-x (read) Other: : r-x (read) Additional groups: rantest/east
Which is clearly on cclinux now.
Further, we can see that east is still checked out, we can modify it and even unco thus showing that our newly relocated snapshot view on cclinux works, from seast1, like it did before:
[p6258c@seast1 p6258c]$ cd p6258c_SIMCQ00000392_intview/vobs/simdev/bin [p6258c@seast1 bin]$ ct ls east east@@/main/simdev_integration/east_scripts_build1_integration/CHECKEDOUT from /main/simdev_integration/east_scripts_build1_integration/0 Rule: CHECKEDOUT [p6258c@seast1 bin]$ echo "A change" >> east [p6258c@seast1 bin]$ ct unco east Save private copy of "east"? [yes] Private version of "east" saved in "east.keep". Loading "vobs/simdev/bin/east" (353 bytes). ClearTrigger removing empty branch "/home/p6258c/p6258c_SIMCQ00000392_intview/vobs/simdev/bin/east@@/main/simdev_integration/east_scripts_build1_integration"...Updating web view with (/home/p6258c/p6258c_SIMCQ00000392_intview/vobs/simdev/bin/east)... Loading "vobs/simdev/bin/east" (353 bytes). Done loading "/vobs/simdev/bin/east" (1 objects, copied 0 KB). Checkout cancelled for "east". [p6258c@seast1 bin]$It would not be that difficult to cook up a script to migrate all snapshot views...
March 27, 2008
EAST Automation
What's been done
- The EAST test automation tool rantest currently can run properly formated EAST tests. By properly formated we mean that they 1) run to completion and 2) report a status in their log file.
- Logfile collection for EAST tests is performed and a tar file, EASTLogs.tgz, is created and if run for record (-rfr) specified that log file will be checked in (coded but untested - see below).
- Many tests can be run in succession using the -file option and specifying multiple tests in the file (See rantest wiki page).
- A rudimentary conditional test execution facility exists. For example, testers can name tests then refer to them in a subsequent test with an "if" statement so as to say "run this test if <named test> passed (See rantest wiki page).
- A facility for running test in the background has been implemented as some EAST test script must run in the background while other tests run simultaneously.
- A crude rendezvous system has been implemented such that rantest will poll the logfile looking for a rendezvous string that will indicate to rantest that the backgrounded test has reached a certain point and it's OK to proceed with other tests.
- Logging of what rantest does has been implemented.
- A crude -script test option has been implemented that allows rantest to "farm out" the verification of a test for tests that require more complex verification. Any script can be coded. The only requirement is that it reports success or failure through it's exit status.
- Worked on learning moshell as some verifications require moshell.
- Implemented a simple case of verifying that a cell is set up (a manual verification step for Sim Cell Setup of preSWIT).
- Ngan has created test profiles and Ross Wilson says that we are going to run these test profiles through rantest this afternoon.
- Documented most of rantest's usage on the wiki.
- Ross and Gantry have created Perl scripts and TPL's such that an EAST test can drive a TM500. While rantest can run such tests it needs to be taught how to gather the TM500 log files produced.
What we're impeded on
- I believe many of the SWIT tests are not currently properly formatted. Such tests will need to be altered before they can be truly automated through rantest
- WRT preSWIT testing, I need Leslie to modify her tests to properly write rendezvous strings in SIM Cell Setup so that rendezvous can work and sanity/preSWIT testing can proceed.
- Gather tests from Ngan (and others)
- We often don't have time on the various RNC's/etc. to perform testing. Some tests, i.e. SIM Cell Setup, are interruptive of other EAST engineer's work.
- Moshell is crude, picky and buggy and... not supported by Ericson!
What's next
- Create run for record view and test out -rfr checkin functionality.
- Get EAST/TM500 test to work (currently failing) so as to check/modify TM500 logfile creation
- Get EAST/NMS test to work (currently non-existant?) so as to check/modify NMS logfile creation.
- Implement XML parsing to determine whether or not we have TM500 logs or NMS logs to collect. Either that or get Gantry to change the format of the message indicating where the logfile is to include what type of logfile it is.
- Once Leslie modifies tests for rendezvous, attempt sanity again and add in the -script "IsCellUp <machine> <cell>" verification.
- Bob wants us to write a simple and a complex example of test verification. Based on Doug Elder's How to Analyze rnc document, implement it's verification steps which get progressively more complex. The IsCellUp script can also serve as an example verification script.
March 17, 2008
Sharable vs Non-sharable derived objects
- Resolved Helpdesk ticket involving non-sharable derived objects
Sharable vs Non-sharable DOs
In Clearcase terms, DOs, or Derived Objects are objects created during a build. These are created when clearmake (or omake) is used. Clearmake will create a configuration record, AKA a bill of lading - basically a record of everything that went into making this derived object. Clearmake then adds a record to the vob saying that it made this DO and if any subsequent clearmake is attempting to make this same DO (given the same input or configuration record that is) then this DO is available for wink in. At this time the DO is copied to the vob and winked into the other view that was requesting this DO be build and the DOs reference count gets incremented. Or at least that's how DOs and winkins are supposed to work
Rational also introduced a concept of express building which essentially creates "non-sharable DOs". A sharable DO and a non-sharable DO are essentially the same except that the advertisement of the DO is not made in the VOB.
Here the error message stated:
'Unable to change meta type of "/vobs/rnc/rnsap/trace_support/decoder_tmp" to create derived object reference count - No such file or directory'.
Notice the "_tmp" portion. The DO was named decoder, not decoder_tmp. I first thought perhaps Clearcase is just having a problem creating decorder_tmp which it may have needed temporarily. I tried touching that file and it was OK. I then removed the decode_tmp file I touch and tried the checkin. It failed again.
The view was marked as nshareable_dos:
$ ct lsview -full -properties p57347_RANCQ00028109_devview | grep Properties Properties: dynamic ucmview readwrite nshareable_dos
This is odd because the default, as shown by:
$ ct lssite -inquire | grep shareable view_shareable_dos=TRUE
is turned on for the site. I then thought that perhaps the problem is trying to check in a nonshareable DO is not allowed. Reading on the ct man winkin page is says:
winkin also converts nonshareable DOs to shareable (promoted) DOs. If you specify a nonshareable DO, winkin first advertises the DO by writing information about it to the VOB, and then promotes it by copying its data container into the VOB and moving its configuration record into the VOB. Because a shareable DO cannot have nonshareable sub-DOs or sibling DOs, winking in a nonshareable DO also advertises its sub-DOs and siblings, converting them to shareable DOs. With -siblings, winkin advertises and promotes the DO's siblings.
However even a ct winkin decoder gave the same error message. So then I again touched decoder_tmp and then ran the ct winkin and it worked! So I then checked in this derived object.
Note: You can change changed a view to sharable_dos with:
$ ct chview -sharable_dos p57347_RANCQ00028109_devview
March 5, 2008
Santity Progress
- Spoke with Tom regarding making a rantools, or perhaps ran_swit_tools and ran_preswit_tools vob. Tom got the helpdesk ticket from Chris English regarding making a preswit vob. I added on the swit vob and discussed with Tom how to best do this.
- Re-ported changes from Rexec version of easter -> non-Rexec version. Looks like we will be having a rantools vob replicated to cclinux. In any event, easter has been redesigned to run locally on seast1.
- Figured out the problem with Rexec (and Expect for that matter). Turns out the Expect problem is the same Expect problem it always seems to be. Expecting a certain string and not seeing it. The twist is that this is indeterministic due to timing and stems from the odd way that start_auto_east operates. You see after executing start_auto_east control gets returned to you, thus you get a prompt, however background processes are also started and they write to the terminal, sometimes interrupting the prompt! This is not good. In general background processes should not be writing to the screen. I need to see if they can turn this off.
- Learned that specifying "-run -log" will run the test but not turn on logging! You must do "-log -run"! In the past sometimes log files would not appear. They would appear if logging was turned on in the profile but not appear if it wasn't. IOW -log was not working because of option ordering! Fixed
- Added rendezvous method and option to that we can run something in the background and then rendezvous with it by searching for a string in the produced logfile.
- Learned that the logfile's presence is not immediate. IOW we start a configuration task, running it in the background, then call rendezvous in order to search the logfile for the message stating it's ready. However the logfile does not exist at this moment. So we must wait. Currently rendezvous uses timeout to wait. It does so by trying then taking 1/10 of timeout and sleeping, proceeding 10 times and then timing out. I need to include the locating of the logfile in this timeout cycle.
- Moved stackOptions and -name, -if and -renedezvous into East.pm and process options using GetOptions. The problem was that I'm coming from a string, the test name followed by test options, and with -if and -rendezvous there's a good chance that the user wants to specify a quoted string. Yet GetOptions only works on @ARGV. So how to get a $str -> @ARGV taking into account that if you hit a quote you need to combine all tokens until the closing quote? Well that's what stackOptions does. So now this is all handled in East.pm. Additionally the test results hash needed to be passed to exec so that exec could do conditional test execution. This makes us pass two hashes and thus need to use references. All this is working now. Need to check to make sure that test results hash is updating in exec and that conditional test execution is working
- Fixed problem in getting logfile name in that $self->{testName} was not getting set properly. Now it is and logfile names are being found
- Still am not seeing what I'm expecting to see in the logfiles for rendezvous. Need to do testing by hand again and save the logfiles to show Leslie/Jonathan.
- Changed to use a hash for all statistics. Added stats for skipped tests and rendezvous tests. This method is much more flexible and counters can be easily added
- Added hack to get around Expect problems
- Determined that messages seen in the text logfile are not already shown in the binary logfile
Expect problems
Last night I managed to discover what was causing my intermittent problem. This problem turns out to be one from Expect but it's more of a challenge. You see whenever you are writing stuff in Expect you are usually participating in a little black art. The challenge often is distinguishing between what you are looking for that you expect and what you get, which is sometimes not what you expect.
In general you are often issuing commands to the underlying process and then searching to see when it is done. So you are often looking for a prompt. The regular expression that I use for standard prompt is:
qr'[#>:%$] $'
For seast1 this generally matches the "$ " that appears at the end of the prompt. There are two problems that can occur: First, output from a command can output exactly what you are looking for ("$ "). Fortunately having a command output a "$" followed by a " " at the end of a line is relatively rare.
The other problem that can happen is when something interferes with terminal I/O when outputting the prompt. How can this happen? Well when background processes indiscriminately write to the terminal at random times. What does this? start_east_auto does! Or rather things that start_east_auto calls does. For example, it calls restart_servers, which does an echo and calls other scripts, which probably also echo stuff.
If any of this background I/O gets written while the prompt is being written there's a chance (and it happens) that my Expect call will never see the prompt and it will timeout. The general rule is if you are processing in the background - don't output anything to the screen!
Can somebody look into this to see if these echo statements can be eliminated or at the very least, sent to a logfile that the user can examine if they want?
Otherwise I cannot guarantee that testing will not occasionally fail because of this.