Status for Andrew DeFaria

Easter

• Paged usage
• Added logfile analysis
• Added Timeout detection
• Added environment variable support
• Changed to use GetConfig for -file

The following progress has been made on easter, the EAST Test Driver:

• easter -usage has gotten bigger so it's now piped to $ENV{PAGER}
• Added log file analysis: easter now examines the log file produced by the running of test cases. It does this by extracting the test case name, not to be confused with the test case file name, from the profile file and composing a (long) path to the log file that the running of East produces. Note that easter is running remotely from a workstation on the RAN and driving tests and processes on raneast and ceast<n>. As such it dynamically expands environment variables like $MNT_DIR and $EAST_REL in composing the path to the log file so that in the event of a change in either of these environment variables will be automatically sense and it will Just Work®.
  
  An additional complexity is that part of the path to the logfile contains a timestamp down to the millisecond. It's difficult (read impossible) to determine the exact timestamp that will be issues and guessing has it's challenges too. Instead, easter takes the approach of using "ls -t | head -1" to determine the most recently modified directory (timestamp directory that is). This works reasonably well provided we don't have multiple tests or processes writing in that directory.
  
  Once the logfile is determined it is examined for "EXECUTION STATUS" and that is reported back to the user.
• Implemented rudimentary detection of timeouts. In testing I noticed that TS_default.profile never exits. So instead it times out. This situation is detected and counted as a timeout (I'd still like an example test case that exits without indicating Success or Failure).
  

  Note: It's been observes that other timeouts have been happening. For example, sometimes the connection to raneast seems to just hang. This could be due to abnormal load on raneast, ceast<n> or other hardware. The easter script does not always handle such things that gracefully yet. In general easter allows for up to 30 seconds for each remote execution of a command, including tests. How long do tests take?

• Added environment variables. The user can now set into the environment various parameters for easter so that they need not have to specify them on the command line. Generally this is used to set things that remain the same for the whole execution of easter like <view>, <type> and <unit>. There is a scheme for overriding that being environment, command line and config file. The environment variables are as follows:
  

  EASTER_VIEW  Analogous to -view  (e.g. export EASTER_VIEW=p6258c_easter)
  EASTER_TYPE  Analogous to -type  (e.g. export EASTER_TYPE=rbs)
  EASTER_CLASS Analogous to -class (e.g. export EASTER_CLASS=ts)
  EASTER_UNIT  Analogous to -unit  (e.g. export EASTER_UNIT=5)
  EASTER_TEST  Analogous to -test  (e.g. export EASTER_TEST=test1.profile)
  EASTER_FILE  Analogous to -file  (e.g. export EASTER_FILE=testsuite.conf)
  

  (and I was real tempted to make an EASTER_EGG!)

  Any options specified on the command line override any options specified in the environment. So the user can do something like:

  $ export EASTER_FILE=testsuite.conf
  $ easter
  

• Expanded the syntax of the -file file. This now uses the GetConfig module. This is a more simplex form of config file, less complicated than say XML. It's more akin to X Resource files and allows for commenting. The config file now can contain configuration options embedded in the file as well as test executions. Comments are supported with the standard "#". Here's an example file:

  ##############################################################################
  #
  # Name:          testsuite1
  #
  # Description:  Test suite file to test easter
  #
  # Author:        Andrew@DeFaria.com
  #
  # (c) Copyright 2008, General Dynamics, all rights reserved
  #
  ##############################################################################
  
  # Use the following view
  view:   p63630_SIMCQ00000607_intview
  
  # Set type and unit to rbs 6
  type:   rbs
  unit:    6
  
  # Execute the following ts tests
  exec:    TS_b2_l3_auto_test.profile
  exec:    TS_success.profile
  exec:    TS_failure.profile
  exec:    TS_profile.profile
  exec:    TS_default.profile
  

  We can see that we are using a specific view, type and unit, which holds for all tests. Each exec line is cumulative (passes in an array to the script) of test cases to run. We can easily execute this "testsuite" then by doing either of:

  $ easter -f testsuite.conf
  

  or

  $ export EASTER_FILE=testsuite.conf
  $ easter
  

  The resulting output is:

  Randws016:ct setview p6258c_RANCQ00023645_devview Randws016:cd /vobs/rantest/tools/lib/Nethawk Randws016:export EASTER_FILE=testsuite.conf Randws016:easter Please be patient while we set up the test environment... done ts      TS_b2_l3_auto_test_.profile Success ts      TS_success.profile Success ts      TS_success.profile Failure ts      TS_success.profile Failure ts      TS_default.profile Timed out easter summary 5 tests run 2 tests passed 2 tests failed 1 test incomplete

Posted by at 5:55 PM | Permalink | TrackBacks (0)

• Initial coding of East.pm
• Added parameters to testeast script
• Changed Rexec to handle timeouts better
• Opened WOR for Rexec work

East.pm Library

Started coding the East.pm library. The idea was that this library would allow testers to invoke tests using a few simple Perl statements. Work on this was initially laid out by Gantry York for other simulators like TM500, etc. This East.pm merely falls in line with those.

I'm trying to code a Perl library for running test cases/suites on East. I'm new to this environment so I don't understand it well. It seems like there are test cases and test suites. There may be "load runner" things and "regression runner" things. All of these objects and how this works is all Greek to me at this moment.

What I'm told is that in order to run these things the following steps must take place:

1. From the RAN, ssh to raneast
2. Once there a TCL/Tk script will pop up a window that shows various RBS/RNC/East things numbered from 1-7. Some boxes will be red indicating that they are in use and some will be green indicating that they are available.
3. I assume I'm supposed to pick a green one. Whether I'm looking for an RBS or an RNC or even an East is unknown to me at this time. This is chosen based on the Device Under Test, if you want to test rbs6 that is what you would choose. East logins are used only for scripting purposes, and will not work for actual execution.
4. So far I've only played with RBSs so I pick a green RBS box and type in "rbs <n>" where <n> is the number of the green box I chose. This apparently goes through a mapping of sorts and ssh's to a ceast machine of appropriate number.
5. Once there I do a start_east_auto <view name> <rbs<n>> where this time "rbs" and "<n>" are squished together. Step 4 above has a space between the "rbs" and the "<n>". This, in some manner, starts the East server(s) and another GUI appears. There are various buttons - mostly initially yellow. I am supposed to wait 10-15 seconds for them (well some of them) to turn green.
6. Next I run java [type of test] -run -name <name of test>.

At this point I'd like to ask some questions:

1. Does the "rbs <n>" command and the "rbs<n>" parameter have to be the same with respect to <n>? Yes, the choice should be the same with regards to the rbs/rnc and The first is used to ssh to the correct ceast blade, the second is used to execute the east application with the correct settings for testing that particular node. The second command is also what creates the lock file indicating you are currently using that resource (the box changes from green to red in the TCL script).
2. How can I programmatically determine if the desired "rbs <n>" unit is indeed green?There is a command line tool called elock that prints the status of each device. The DUT name will only be present if it is currently locked.
3. What is [type of test]? From the Command Line in EAST document I have ascertained the following "types" of tests:
   • LoadTCRunner The type of test which is run is called a Test Case or “TC”, it is run in Load which means it can be run with multiple contexts.
   • RegressionRunner Same as above, except the test is run as a single context
   • RegressionLoadRunner Not used in our current implementation
   • RegressionTSRunner This command runs Test Suites or “TS”

From what I can tell, and from an IM with Ross, there appears to be the following relationship:

Java Command	-name	Path	Comments
LoadTCRunner	<test>.profile	/simdev/tc_data/tc/profiles/load
RegressionRunner	<test>.profile	/simdev/tc_data/tc/profiles/tc
RegressionLoadRunner	???	???	Not used?
RegressionTSRunner	<test>.profile	/simdev/tc_data/ts/profiles/ts

Note: Path is important as you cannot specify -path to the java command.