LOS178-570/int/package and check

  • Attempting to build LOS178 - CR 570 (Posix)
  • Tried to make changes for package.sh. Problems are:
    1. Waiting for CR fom Moscow to address including of libstdc++ for LOS 2.1.0
    2. GD lacks libstdc++ from ppc.cdksol.tar.gz! Entered CR #38 for that
    3. It was thought that changes to package.sh could be made without the need for a CR. Unfortunately the CVS setup is such that a CR is required, even for comitting changes to toolbox
  • Described new scripts for check and package. Check can be used to help tally the number of growing warnings
  • Described plan for a /int area so as to centralized our tools and scripts

Check and Package

I've done a little work on trying to come up with more generic scripts that can be used across build machines or indeed the organization. Specifically I've been targeting things like packaging and checking. My scripting language of choice is Perl.

All scripts are relative to my machine at present at saturn:/int. You are free to mount this file system from my machine to your machine to check out the code. In the future I plan on putting this all into a cvs repository and making if available to all who want/need it via a global file system such that adding /int/bin and perhaps /int/adm/bin to one's PATH is all that is needed to gain access to these scripts on any machine. Details of this will be forthcoming in another email.

In this message I will only describe the check and package scripts. Other scripts, including such things as cvs_report, ecrc as well as web pages and scripts will eventually be documented probably through a series of web pages.

Check script

Jas had asked me to enhance the checking of install.logs so as to keep track of the growing number of warnings. The check script does this. It will check for errors using the common error strings of other check like scripts as well as warning and optionally issue a total number of errors and warnings. The check script is in /int/bin/check. Here's a short usage:

    Usage: check [-u] [-v] [-d] [-t] [-w] 

    Where:

            -u              Display usage
            -v              Turn on verbose mode
            -d              Turn on debug mode
            -t              Display total line
            -w              Include warnings
                   One or more log files to check

With no options except logfile(s) check will output nothing but set the return status to the number of errors encountered. This allows future script to be able to use this in their script and just check the return status.

Package script

The idea here is to separate the definition of a package from the packaging code itself. The hope is to hand the package definition over to the developers themselves so that they can maintain that - after all they know better what goes where than we do.

Packaging is implemented as a module, specifically a Perl module and as a Perl object itself. The idea is to encapsulate what can be done with a package into an object. The object module is at /int/lib/LWPackage.pm. The "LW" stands for "LynuxWorks". An LWPackage object currently has 3 methods: new, list and package. The new method is called to create a new package object. One parameter must be specified, that being a pathname to a package spec file. That package spec file is parsed and the package object is populated. The list method will list all of the information about the package including the file list. Finally the package method produces the package image itself.

Here is a small Perl snippet that utilizes the LWPackage methods:

    my $pkg = LWPackage->new (spec => $spec);
    $pkg->list if get_verbose;
    $pkg->package;

In fact that's the main code for /int/bin/package, the packaging script. It creates a new LWPackage using the filename in $spec, calls the list method if verbose is turned on (get_verbose is part of the Display package) then calls the package method to create the package image.

Package spec file

Package spec files are denoted by the .lwp extension convention. The format of the spec file is pretty simple. As usual "#" indicate comments, etc. Look at /int/spec/* for example files. Basically the format is similar to:

    Name:       int
    Version:    1.0
    ProductNbr: 1000
    Release:    00
    Base:       /int
    Fileset:
        *
        -data
    EndFileset

Name, ProductNbr, etc. are used in the creation of the image file name ($ProductNbr-$Release.$Name.$Version.tar.gz). The Fileset section defines the file sets included in the package relative to Base. An "*" denotes the usual connotation of "everything". Other path names could be listed one by one. For example, I could have:

    Fileset:
        adm
        bin
        spec
        -data
        -web
    EndFileset

A minus sign does what you'd think - remove files denoted by this. So the above says, "everything under $Base except everything under $Base/data".

As for Fileset lines you can list multiple lines and they are processed in the order that they are entered. Duplicate filenames are removed and the Fileset list is ultimately sorted. You can list either directory names for whole directories (that are recursively processed) or individual file pathnames. Regex's are not supported (yet but should be).

The idea of a package spec is to define the Fileset from where they stand thus eliminating the need to copy large quantities of data into an alternate area so that that alternate area is "clean". IOW it should be able to pull only those necessary files from even a CVS or build tree directly. To that extend future arrangement of things into distinct areas will make writing packaging specs easier.

A plan for /int

We are seeking an NFS mountable global area where we can place various tools and scripts that will help us do our job. A previous email about the check and packaging scripts are an example of this. The basic idea is to be able to mount a global file system area to a short named path on the local system so that scripts like package and check, etc are accessible and readily available when needed.

So, for example, if t3:/export/int were that globally accessible file system one would mount t3:/export/int /int and then add /int/bin and perhaps /int/adm/bin to their path and these scripts would be available.

As these scripts will be used in business processes they are as valuable as our products themselves and thus should be placed under CVS. Let's assume that t3:/cvs/int-cvs were the CVS repository for these scripts. We could check out and modify our own scripts as we further develop them. A nightly cronjob could be set up to cvs export -r >RELTAG< /export/int to make sure that /export/int reflects where >RELTAG< is some release tag. Then people could check out and enhance/modify our scripts in a test like environment and when "released" simply move the >RELTAG< to the committed version. The updated version would be available the next day (or we could force it by hand if necessary).

As for what gets put into /int I see the following directory structure:

adm: Administrative area
bin: Administrative bin scripts
data: Any adm data files that you might need
etc: Rough equivalent of /etc
functions: bash script functions (currently symlinked to ../functions)
bin: Scripts and apps for int
data: Any data you might want
functions: bash script functions
lib: For Perl and other library modules
spec: Spec files (may get rid of this)
test: Any test scripts. For example, test script to test the functionality of the modules in ../lib
web: Might want to move this elsewhere. Basically a replica of the web pages and scripts I've been doing (so as to have a copy)

Of course this can change and evolve over time. The main idea is to have a standard place that is globally accessible and short pathed (you could easily type /int/bin/check if /int/bin is not in your path) at a well known path name. Also to sort of replicate or mimic the OS's standard directories like bin, etc, and the like so that it's easily understandable and "natural".