More documentation
- Yet more documenting of Build, Test, Release processes
« September 2004 | Main | January 2005 »
Directories, builds, dated directories, etc are all over, spead out between t3, dopey, t-k1g-1, etc.. It's hard to know where things are and where they need to end up. Will look into this tomorrow with Thu
Regarding cmptags.pl, rearranged this to attempt to produce a simple list like:
ECR Number ####
path/to/foo.c 1.30
path/to/bar.c 1.2.3.1
Unfortunately this appears to be very slow to process. cmptags.pl does not attempt to optimise things only working on files recently updated, rather it goes though all of the files in CVS and generates, parsed and creates data structures for all the CVS info, even that which it's not particularly interested in at the momment so that it can be more generally used. This takes forever with CVS as each file must be found and a cvs log issued and output parsed. Unlike Clearcase there doesn't seemto be a cvs command line prompt so cvs is constantly being invoked (process overhead) and it's always talking to the server (network overhead) and there's a lot of data (tags, revisions, comments, etc) to be processed. Clearly CVS has not been optimized for this sort of access.
Not sure how or if this can be optimized any further. It's a shame because I'm almost at the point where I can ask "Show me the files and version numbers for the following ECR's" which is something I know I'm gonna need quickly here for the impending build note that needs to be created.
First I obtained the latest TOT for los178 - apparently they are still fixing things. This makes it imperative that fix_copyright.pl work flawlessly. Was able to build los178 from old, unmodified sources thus I know I'm starting from a good set of sources.
Next was to run fix_copyright.pl and attempt a compile. This time, instead of just hand editing the files I examined the file in question and why fix_copyright.pl failed in this case. Then fix fix_copyright.pl and repeat.
One problem was that some files have MS-DOS line endings. Fixed this by removing such line endings and making them standard Unix line endings.
Next some files ended the comment block with */ and then some additional spaces. This was fixed by not attempting to anchor the end comment block match with the end of the line (i.e. $).
Injected these actual two or three files back into the build process until it built.
For safety's sake I then destroyed my build area and recreated from scratch with cvs update -C then running fix_copyright.pl again and bringing that over to grumpy.
WRT cmptags.pl, made some more progress parsing cvs log info...
Generally these copyright blocks with two copyrights are of the form described yesterday, with a single copyright line added for LynuxWorks. Such files will not be modified
Need to work out remaing los178 build problem and make fix_copyright handle them too.
Tried to get a Perl CVS module instead of coding it myself - seems there are no complete Perl CVS modules on CPAN!
I'm finding it very difficult to develop a hueristic algorithm to find what I call "LynuxWorks copyright blocks" in existing source files. The problem is that the input data is not necessarily consistant. Sure we can go for the 80% fix and hand fix the rest (in fact, that's what I did before) however those 20% often cause compiler errors during the build. What's potential worse is that it is possible that some files will be altered in such a way as to not produce a compiler error but rather to change behavior of the resulting code.
My current algorithm attempts to do the following:
Currently the "official" copyright block is of the form:
/* vi: ts=4 sw=4
*************************************************************
(C) Copyright $copy
LynuxWorks, Inc.
San Jose, CA
All rights reserved.
$Date: $Date$
$Revision: $Revision$
$Source: $Source$
************************************************************/
The "$copy" is replaced by the copyright date found in the file. Dates of the form
/* Start Copyright ******************************************
vi: ts=4 sw=4
(C) Copyright $copy LynuxWorks, Inc.
San Jose, CA
All rights reserved.
$Date: $Date$
$Revision: $Revision$
$Source: $Source$
* End Copyright *********************************************/
This would 1) retain the "vi: ts=4 sw=4" annotation that I assume is for vi users, 2) group both the copyright string ("(C) Copyright") along with the company name of LynuxWorks on the same line. This makes it easier to grep for in the future considering some files have Rockwell copyrights. 3) Clearly delineates the start and stop of the copyright block.
The problems that I'm having is that I'm seeing copyright blocks of the following forms:
/************************************************************
(C) Copyright 1987-2000
Lynx Real-Time Systems, Inc.
San Jose, CA
All rights reserved.
$Date: 2003/11/14 22:44:44 $
$Revision: 1.1 $
************************************************************/
(Does not contain LynuxWorks, rather Lynx Real-Time Systems)
/*
.FP
***********************************************************************
Revision History
See ClearCase
Version:
***********************************************************************
*
* EXPORT NOTICE:
*
* INFORMATION SUBJECT TO EXPORT CONTROL LAWS
*
* Subject to local country rules and laws when applicable, you
* must comply with the following:
*
* These commodities, technology, or software were exported from
* the United States in accordance with the Export Administration
* Regulations. Diversion contrary to U. S. law and other relevant
* export controls is prohibited. They may not be re-exported to
* any of the following destinations without authorization; Cuba,
* Iran, Iraq, Libya, North Korea, Sudan or any other country to
* which shipment is prohibited; nor to end-use(r)s involved in
* chemical, biological, nuclear, or missile weapons activity.
*
* COPYRIGHT NOTICE:
* (C) Copyright 2001 Rockwell Collins, Inc. All rights reserved.
*
* FILE NAME:
* df.c
*
* PURPOSE:
* utility to display disk usage
*
* NOTES:
*
* ABBREVIATIONS/ACRONYMS:
*
*****************************************************************
.FP END
*/
/************************************************************
(C) Copyright 1987-1996
Lynx Real-Time Systems, Inc.
San Jose, CA
All rights reserved.
$Date: 2003/09/10 15:24:57 $
$Revision: 1.1.1.1 $
************************************************************/
Contains multiple "(C) Copyright" strings, one being ours and the other being Rockwell's. Should both exist in the resultant file?
/*
.FP
**********************************************************************
*
* FILE NAME:
* hm_load_header.c
*
* PURPOSE:
* Performs the integrity check of the program and data files
* in the CPR read only file system.
*
* ABBREVIATIONS/ACRONYMS: (optional)
*
* NOTES: none
*
* COPYRIGHT NOTICE:
* (C) Copyright 2001-2002 Rockwell Collins, Inc. All rights reserved.
* Proprietary and confidential material. Distribution,
* use, and disclosure restricted by Rockwell Collins, Inc.
* Copyright (c) 2003-2004, LynuxWorks, Inc. All Rights Reserved.
*
***********************************************************************
.FP END
*/
Contains multiple "(C) Copyright" strings, one being ours and the other being Rockwells, in the same comment block! Also notice the inconsistant form of one bying "(C) Copyright" while the other being "Copyright (C)". How should this case be handled?
The Perl script to fix the copyrights in LOS178 sources needs detect if there was a prior copyright block and replace it if necessary. Finding old copyright blocks is hueristic in nature and thus fallable. The problem is there are no sentenals clearly marking our copyright. The best the script can do is attempt to find a comment block that starts and ends with a particular comment line based on the "/" and a number of "*"'s for the start and a number of "*"'s and a "/" for the end. Problem is we do not know if what's inbetween is really our copyright. The hueristic can be tuned finer with some difficulty but currently it does not.
The real difficulty is that there is no real standard in our copyright blocks thus it is hard to get this right
As it stands the script occasionally misinterprets a comment block, sometimes just the ending comment block, and alters the file in such a way that it introduces compile errors.
There were a number of files that this occurred in and they were hand edited
In picking up ECR 23084 I was able to build the toolchain for both cross x86 and ppc. In building the native toolchain there is an error:
Configuring texinfo... configure: error: invalid package name: gcc-version-trigger Configure in /mnt/toolchain/build-i386/texinfo failed, exiting. make: *** [stamp-configure-i386] Error 1
Thu says that I didn't run fixup.sh but I did. The toolchain does not package up fixup.sh so I grabbed one from somewhere. Thu suggests that we file an ECR for this and fix the Makefile.
Thu determined that we have a toolchain problem here so she had me submit ECR 23079. Adam debugged the problem which produced 23080
Failure reason : Compiler does not exist: /export/dev_archive/lynxos/tools-5.0.0/3.2.2-072304/toolchain-powerpc-lynx-lynxos-ppc.tar.gzAh ha! See this is one of those things that just confuses me. Contained in __profile.exp there's:
# compiler binary
#set DIR_PATH(compilers) "/usr/lynx/t3/dev_archive/lynxos/tools-5.0.0/3.2.2-072304"
set DIR_PATH(compilers) "/export/dev_archive/lynxos/tools-5.0.0/3.2.2-072304"
need to uncomment the first line and comment out the second line.Thu said that the error sin makeall.log were acceptable. Continuing on to build the ppc toolchain...
$ rcp toolchain-i686-pc-linux-gnu-powerpc.tar.gz int@t3:/export/dev_archive/lynxos/tools-5.0.0/3.2.2-120604
set DIR_PATH(compilers) "/usr/lynx/t3/dev_archive/lynxos/tools-5.0.0/3.2.2-120604"The compiler exists there (because of the archiving step above). Note there was no t3 style reference here.
I was under a mistaken assumption that I needed to have a PPC machine in order to do this and was waiting for one. Thu informed me that I could build 5.0.0 for cross and even rebuild the toolchain cross - that I wouldn't need a ppc machine until I needed to build a native gcc.
Building for ppc takes a long time as our ppc machines are slow and there's a lot more to build for ppc.
Failed to get loadit.exp to work. Seems that it is oriented to getting the images off of t3. But I didn't put this new images on T3 yet. Seems there's some rync process that Thu flew by me and I missed.
Note: loadit.exp requires a working network connection to the System Under Test! When last we left t-k1g-1 it was booted to lynxos with non-working networking. loadit.exp sits and waits at an rcp command and the SUT never responds!
Came in to find that the native build of the 5.0 toolchain went nowhere. Seems, as I expected, that I need to unpack 18010.x11r6.tar.gz and 18011.pd.tar.gz too. Unpacked that and restarted build.
Build failed again, this time Memory Exhausted. I had to do ulimit -s -d 1000000 and restart build
Build finally succeeded. Did make package then copied to t3 area (to be precise t3:/export/dev_archive/lynxos/tools-5.0.0/3.2.2-120604). Only needed to copy over the toolchain-i386-lynx-lynxos-i386.tar.gz as this package is the 5.0.0 toolchain built natively
Created a new build area on dopey @ /export/build1/LYNXOS_500/build/lynxos/120604-C. Needed to copy over int_tools and modify int_tools/__profile.exp to:
Note: I find all of these different machines with different long pathnames confusing. For example, I would think using the principal of separation of code and data that the int_tools code should be globally accessible and that one should not be modifying variable contents before running scripts. That's data and should either have to be supplied or kept in a data file perhaps separate for each build run. IOW int_tools should be globally visable (perhaps in one's PATH) and execuable as code and not modified per run. Also, as such it should in CVS and perhaps grabbed from CVS with an official TAG.
Managed to build the toolchain (gcc 3.2.2) successfully. We had to first build LynxOS then build the compiler then build LynxOS again. Confusing indeed. Lot's of paths and scripts. I'm still not totally comfortable with the environment, how to build, native vs. cross, toolchain vs. LynxOS, CVS top of trunk, etc.