1/31
- Added html capability to cvs_report.pl
" />
« December 2004 | Main | February 2005 »
Vinnie pointed out that there was some sort of problem with CVS on Monday, the date that I had gotten the sources out of CVS. Ends up some files were missing so I had to redo the PPC cross build.
I've updated cvs_report.pl to be more parameterized and flexible. One goal was to make it smart enough to produce a report showing the ECRs between two labels. This is helpful when one wants to get the ECRs that were picked up in this build.
Here's the usage:
$ cvs_report.pl -u Usage: cvs_report.pl [-u] [-email-to <list>] [-email-cc <list>] -from <tag> -to <tag> -m <cvs_module> Where: -u: Display usage -email-to: Comma separated list of email addresses to send the report to (Default: Report sent to STDOUT) -email-cc: Comma separated list of email addresses to cc the report to -from: CVS tag used as the base (Default: Extracted from CVS area - error if no CVS area) -to: CVS tag used as the delta (Default: HEAD) -m: CVS module to process
Notice that there are options for -email-to and -email-cc. If no email options are specified then cvs_report just outputs the report to STDOUT. This can be used to capture what was picked up. For example, to pickup the ECRs that were incorporated between Lion_lynxos_120604 and Lion_lynxos_012405 one could:
$ cvs_report.pl -from Lion_lynxos_120405 -to Lion_lynxos_012405 -m lynxos > /tmp/ECRs.log
Note that if -to is not specified then it defaults to HEAD. So the following cronjob could be configured:
* 4 * * * /it/bin/cvs_report.pl -email-to <email alias>
(It would be best if email aliases were used to control who the report is normally sent to).
You may notice that there is no -from in the cronjob. This is possible because of another optimization change to cvs_report. The old cvs_report.pl would check out a module into a temporary directory based on a tag then do an update capturing and processing the output from cvs update as the files that have changed. Then it would remove that temporary area. This is a waste as the next night the CVS area is created again from the same -from or base tag. The new cvs_report will also create the temporary area by using $module.cvsr (e.g. lynxos.cvsr). Additionally it will create a file in $module.cvsr named .cvsr containing the -from tag. Subsequent runs of cvs_report without a -from will use the .cvsr file to ascertain what base level the CVS area is at. Thus it does not need to "refresh" the CVS area again and can instead proceed to the cvs update step. This saves several minutes of processing time.
Finally, even the cronjob does not need to ever be changed (except if we change the -email-to parameter, which, again, should be an email alias thus we would not change the cronjob rather we would change the email alias - think of people subscribing to receive the CVS report). When we go from Lion_lynxos_012405 -> the next tag all that need be done in a shell window is:
$ cvs_report.pl -from <new tag> -m lynxos
Since -from is specified, cvs_report will refresh the lynxos.cvs area and update the .cvsr file to reflect the change. The nightly cronjob will automatically sense the change and just work!
Oh, and in passing, I make a change such that new files, files who were not present at the -from tag, are tagged with an "*" to indicate that this file is new file in CVS.
The LynxOS I put on t-mcpn765-1 was suspect as I had untarred a couple of different X/Motifs onto the root file system and the toolchain build was failing anyway. This was a good opportunity to redo this and make sure my documentation was correct.
Note: For this building of the toolchain I used the 20210.xfree.tar.gz for the first time
To build the toolchain I did the following (note that I had already got a copy of the toolchain src and unpacked it onto the k partition):
# . SETUP.bash Setup : ENV_PREFIX is an environment variable that points to : the root of the build/release directory. The ENVIRONMENT : file is found as $ENV_PREFIX/ENVIRONMENT. : Set ENV_PREFIX to /? [y] Target : Please specify the target (ppc, x86, arm or mips) : ppc Setup : Complete! # ulimit -s 1000000 -d 1000000 # mount /dev/sdncr.0k /mnt # cd /mnt/toolchain/3.2.2/010405/ # ls Makefile fixup.sh* src/ build-powerpc/ install.log stamp-configure-powerpc # make clobber rm -rf {build,install,stamp-*}-powerpc # ./fixup.sh + cd src + touch texinfo/intl/plural.c + touch libgui/library/tclIndex + find . -name aclocal.m4 -exec touch '{}' ';' + find . '(' -name config.in -o -name config.h.in ')' -exec touch '{}' ';' + find . '(' -name Makefile.in -o -name stamp-h.in ')' -exec touch '{}' ';' + find . -name configure -exec touch '{}' ';' # make install > install.log 2>&1 &
Thu helped me recover from the makeboot -r 0,48 preboot. This LynxOS 5.0.0 has not networking so I booted back up to the b partition (LynxOS 4.0.0) and mounted the c partition. Next I got X/Motif from t3:/export/dev_archive/lynxos/xmotif/032504 (note that this tarball was named 20010.xfree86.tar.gz) and unpacked it onto the c partition. It is important that XFree is unpacked into the partition and root diretory where LynxOS is.
I had already copied toolchain-src.tar.gz over to the k partition and unpacked under toolchain/3.2.2/010405
Next boot up back to the 5 partition and mount the k partition. SETUP.bash for "/" then go to toolchain/3.2.2/010405 and make install...
Attempting to learn and document how to load the LynxOS manually. Normally the int_tools handles this process but occasionally we need to do this by hand.
Messed up by not following all the required steps. Had to get a CD from Hardev in order to get past a preboot problem. Managed to do that.
The instructions I was given to follow for manually loading LynxOS were x86 based. I am expected to translate that to PPC. In any event, the makeboot -r 0,48 preboot is wrong for this PPC machine. This has caused me to have an unbootable system once again. How does one know what numbers to use for a given system?
Reviewed documentation I've written so far, mostly just Cross LynxOS build. Several suggestions were made and being implemented.
Thu's notes simply said "/dev/sdncr.0c -> load the 5.00. tot build". I was not sure how to do this. I did notice that I had never rynced my last PPC LynxOS build over to the T3 area so I did that. Still I'mnot sure how to "load the 5.0.0 tot build" onto a machine. My guess is:
My fear is that the devos will not fix and/or I need other tarballs than just 20029.bsp_mcpn765.tar.gz. Also, just laying this on top of root seems wrong, what happens to the old 4.0.0 stuff?
The above is not really true. The process of manually loading and OS is more complex. It is codified into various expect script and detailed in "install logs" which appear under int_tools/INSTALL_LOG. Worked on documenting this manual OS load.
Although some errors persisted in the 120604-C build I decided it was time to at least try a native PPC build of the toolchain. The problem here is the PPC machine that I was told to use, t-mcpn765-1, is a LynxOS 4.0.0 machine! I don't think I'm supposed to try to build a 3.2.2 toolchain for 4.0.0.
Additionally, I've never built the toolchain natively on a PPC machine before. Previously, when building the toolchain natively, one needed to extract things like X/Motif (according to my instructions - which may very well be off base - why would we need X/Motif to build a compiler?!?) and devos (I think). Where do I get such things?
Additionally, t-mcpn765-1 doesn't have enough disk space.
After building LynxOS 120604-B and only receiving a few errors about (see previous days entry) I decided to attempt to rebuild the toolchain again. This seemed to work OK!
With the toolchain rebuilt I went to rebuild LynxOS 120604-C. At this point Dopey ran out of disk space. I moved the 120604-A subdirectory over to /export/build2/defaria temporarily because that filesystem had much more space.
There's been this nagging problem of int_tools not being able to find the proper toolchain tarball to install compilers. This we due to the fact that I've been copying int_tools from 120604-A -> 120604-B -> 120406-C. Well names have changed but the copies of int_tools didn't reflect that change! Plus, for some reason, somebody hand fixed 120604-B but 120604-C and -A had been refreshed from CVS. Moral of this story is to always use cvs checkout to get a fresh copy of int_tools.
Note: This underscores one of the things that I do not like in CVS, that being the tendency of making many copies of what's in CVS due to it's snapshot view/sandbox like methodology. It's easy to see where one might create a 120604-C/int_tools directory via a cvs checkout int_tools and that int_tools be different than 120604-B and 120604-C's copies. If one then needs to make a change to int_tools and this change should apply to the -B and -C areas as well then one needs to remember about the -B and -C areas and remember to update them as well. This is a common problem with a snapshot/sandbox philosophy. Nevermind that it's wasteful to have so many multiple copies of essentially the same thing - wasteful in terms of diskspace and prone to misunderstanding and human error.
Adam helped me figure out that the reason that the toolchain was failing to build was because the ppc/lib directory in 120604-B didn't have the proper libraries. I proceeded to build the tookchain using 120604-A.
After successfully building the toolchain in 120604-A I proceeded to attempt to build LynxOS in 120604-B utilizing int_tools. The first problem that I had was that it attempts to install a compiler from DIR_PATH(compilers), which was set to 3.2.2-120604. But the toolchain I built for PPC was not there. Note that there was also a 3.2.2-122804 directory. I decided to use 3.2.2-122804 instead of 3.2.2-120604 and I scp'ed the toolchain I built over to that directory. The 3.2.2-122804 already had some toolchains in there but didn't have PPC toolchains.
After performing a make package for the toolchain I wound up with a toolchain-i686-pc-linux-gnu-powerpc.tar.gz. So I copied that over to 3.2.2-122804. But int_tools was looking for a toolchain-i-686-pc-linux-gnu-ppc.tar.gz. I solved this problem by symlinking. I also needed a toolchain-powerpc-lynx-lynxos-ppc.tar.gz or int_tools complained.
After fixing those problem I managed to build LynxOS but there were a few errors:
LOGS/makeall.log:make[5]: *** [/usr/lynx/5.0.0/ppc/sys/family.ppc/lib/libdrivers.a(flopdrvr.o)] Error 1 LOGS/makeall.log:make[4]: *** [/usr/lynx/5.0.0/ppc/sys/family.ppc/lib/libdrivers.mpc8260_vads.a(flash_mtd_vads.o)] Error 1 LOGS/makeall.log:make[3]: *** [common_all] Error 1 LOGS/makeall.log:make[4]: *** [/usr/lynx/5.0.0/ppc/sys/family.ppc/lib/libdrivers.rpxl823.a(flopdrvr.o)] Error 1 LOGS/makeall.log:make[4]: *** [/usr/lynx/5.0.0/ppc/sys/family.ppc/lib/libdrivers.vmpc.a(flopdrvr.o)] Error 1 LOGS/makeall.log:make[4]: *** [/usr/lynx/5.0.0/ppc/sys/family.ppc/lib/libdrivers.vmpc.a(if_dec21040.o)] Error 1
There are also some errors in demo.log.
Are these reasonable errors? Should I go one to attempt to build the PPC toolchain again using this LynxOS?
Adam correctly pointed out that 120604-B had relatively nothing in $ENV_PREFIX/lib. I don't know what that's so empty but 120604-A had a more full $ENV_PREFIX/lib so we switched over to using that.
The mysterious rsync process has been found! Apparently the int_tools START process cannot write directly to T3's dev_archive area. So, instead it puts it to DIR_PATH(archive) which needs to reside on dopey and then an rsync script, in ~int/toosl/rsync_500.sh, to copy the files over to the official dev_archive area on T3
The psuedo archives, AKA The Dopey Archvies are pointed to by DIR_PATH(archive), typically at /export/build1/LYNXOS_500/archive.
Added more structure to the documentation by including things I've developed at home with PHP. Mainly added a menu on the left hand side so as to allow easier navigation of the pages.
After Moscow fixed the atoi error I believe that the TOT build succeeded!
How does one determine if a build it successful? I ask because sometimes some errors are considered OK. But I don't know how to determine whether an error is acceptable or not. Here's the errors that I'm getting in the TOT build after Moscow has fixed the atoi error:
LOGS/demo.log:make[1]: *** [hello_world] Error 1 LOGS/demo.log:make[3]: *** [demo] Error 1 LOGS/demo.log:make[2]: *** [demo/demo] Error 2 LOGS/demo.log:make[1]: *** [local] Error 2 LOGS/demo.log:make[1]: *** [hello_world] Error 1 LOGS/demo.log:make[3]: *** [demo] Error 1 LOGS/demo.log:make[2]: *** [demo/demo] Error 2 LOGS/demo.log:make[1]: *** [local] Error 2 LOGS/makeall.log:make[2]: *** No rule to make target `install'. Stop. LOGS/makeall.log:make[2]: *** No rule to make target `install'. Stop. LOGS/makeall.log:make[2]: *** No rule to make target `install'. Stop. LOGS/makeall.log:make[2]: *** No rule to make target `install'. Stop. LOGS/makeall.log:make[2]: *** No rule to make target `install'. Stop. LOGS/makeall.log:make[2]: *** No rule to make target `install'. Stop. LOGS/makeall.log:make[2]: *** No rule to make target `install'. Stop. LOGS/makeall.log:make[2]: *** No rule to make target `install'. Stop. LOGS/makeall.log:make[2]: *** No rule to make target `install'. Stop. LOGS/makeall.log:make[2]: *** [rkjump.o] Error 127 LOGS/makeall.log:make[2]: *** No rule to make target `install'. Stop.
I figure the demo.log errors may be acceptable and I remember the [rkjump.o] error. But are those "No rule to make target 'install'"'s OK?
Well I played around with this a little more and came up with a Perl script that will dump ECR descriptions fairly easy. From what I understand that's mostly what we want access to from a Linux box (though I could envision wanting other things perhaps in the future). The problem as I see it is that this script will only run on lynx12. It should be runnable from any machine really however you would need to install the DBD module for Informix for Perl access. Unfortunately this requires at least an Informix Client SDK and that's not free! :-(
Here's the simple script (currently at lynx12:/tmp/ecrdesc.pl):
#!/usr/bin/perl ################################################################################ # # File: ecrdesc # Description: This script will dump out the description for the ECR #(s) # passed in. # Author: Andrew@DeFaria.com # Created: Fri Jan 7 15:35:13 PST 2005 # Language: Perl # # (c) Copyright 2005, LynxWorks Inc., all rights reserved # ################################################################################ use strict; use warnings; use DBI; my $DB; # Called when a database error has occurred sub DBError { my $msg = shift; my $statement = shift; print $msg . "\nError #" . $DB->err . " " . $DB->errstr . "\n"; if (defined $statement) { print "SQL Statement: $statement\n"; } # if exit $DB->err; } # DBError # Connect to database. Note this is using anonymous access (read only) $DB = DBI->connect("DBI:Informix:lynxmigr1") or DBError "Unable to open database"; # Loop through ECR #s from the command line foreach my $ecr (@ARGV) { print "ECR #: $ecr\n"; my $statement = "select description from defect where pkey=\"$ecr\""; my $sth = $DB->prepare ($statement) or DBError "Unable to prepare statement", $statement; $sth->execute () or DBError "Unable to execute statement", $statement; # Defect records are unique per pkey (AKA ECR) there for there will # only be one entry in @row. Also the description is returned as one # large string. my @row = $sth->fetchrow_array; if (!@row) { # @row is empty if there was no ECR by that number print "Nothing found!\n"; } else { my $desc = pop @row; print "Description:\n" . "-" x 80 . "\n" . $desc . "\n" . "-" x 80 . "\n"; } # if } # foreach $DB->disconnect; exit;
From the gnu.cvs.help newsgroup I learned that doing a cvs log of everything then grepping through that is much faster than interrogating each file with cvs log. I then reorganized files4ecr to work in the following manner:
It occurred to me that often we wish to know, or pull, the file revision associated with an ECR #. I've created a script called files4ecr that accomplishes this:
saturn:files4ecr-u Usage: files4ecr [-v] [-d] [-l] [-x] [-u] <ecr> Where: -v: Turn on verbose mode (Default: off) -d: Turn on debug mode (Default: off) -l: Local directory only, no recursion -x: Turn on execute mode (Default: off) -u: Dsplay usage ecr ECR number to search for
The performance for this script is pretty good, depending on the amount of information in the CVS logs. Here's an example of it's usage:
saturn:files4ecr 20505 SETUP.csh: 10.2 - Out of date etc/tconfig: 10.2 - Already up to date SETUP.bash: 10.6 - Out of date etc/ttys-arm: 1.1 - Already up to date Makefile: 10.13 - Out of date
In noexecute mode it just displays the file and the revision of the file associated with that ECR. The -l option is similar to cvs' -l option:
saturn:files4ecr -l 20505 SETUP.csh: 10.2 - Out of date SETUP.bash: 10.6 - Out of date Makefile: 10.13 - Out of date
Also, like cvs, files4ecr operates in the current context and current working directory.
With -x turned on files4ecr will perform the cvs update commands necessary to "pull" the versions for the ECR:
saturn:files4ecr -x 20505 cvs update -r10.2 SETUP.csh - Updated cvs update -r10.2 etc/tconfig - Already up to date cvs update -r10.6 SETUP.bash - Updated cvs update -r1.1 etc/ttys-arm - Already up to date cvs update -r10.13 Makefile - Updated
And we can see that the update has taken place:
saturn:files4ecr 20505 SETUP.csh: 10.2 - Already up to date etc/tconfig: 10.2 - Already up to date SETUP.bash: 10.6 - Already up to date etc/ttys-arm: 1.1 - Already up to date Makefile: 10.13 - Already up to date
Anybody interested in such a script?
Note: I've noticed that "ECR Number:" is not necessarily a consistent indicator of an ECR number. I search for several strings:
/^ECR Number: (\d*)$/ or /^ECR# (\d*)$/ or /^ECR # (\d*)$/ or /^\s*ECR (\d*)/
I've also noticed that a single ECR number may be attached to several revisions. files4ecr takes the latest revision in such cases.
Preliminary investigation shows that while attempting to configure libiberty the build process is unable to use gcc to compile things. It seems to be lacking a crt1.o
The algorithm that files4ecr.pl uses is indeed slow. Combing through lynxos looking for files involved in ECR 20591 took:
real 416m30.889s user 6m43.900s sys 1m42.770s
Yikes!
Here's the plan:
Seems the individual files for this ECR have already been applied to the work area so I just need to build. Here what I did:
Is there a known problem with TOT build? Cause I'm getting an error saying essentially that "atoi is undefined".
gcc -g -O2 -o named aclconf.o client.o config.o control.o controlconf.o interfac emgr.o listenlist.o log.o logconf.o main.o notify.o query.o server.o sortlist.o tkeyconf.o tsigconf.o update.o xfrout.o zoneconf.o lwaddr.o lwresd.o lwdclient.o lwderror.o lwdgabn.o lwdgnba.o lwdgrbn.o lwdnoop.o lwsearch.o unix/os.o ../../ lib/lwres/liblwres.a ../../lib/dns/libdns.a ../../lib/isccfg/libisccfg.a ../.. /lib/isccc/libisccc.a ../../lib/isc/libisc.a -lnsl /export/build1/LYNXOS_500/build/lynxos/120604-C/x86/lib/libnsl.a(getgrent.as.o): In function `getgrent': getgrent.as.o(.text+0xb4): undefined reference to `atoi' /export/build1/LYNXOS_500/build/lynxos/120604-C/x86/lib/libnsl.a(getgrent.as.o): In function `fgetgrent': getgrent.as.o(.text+0x2b8): undefined reference to `atoi' /export/build1/LYNXOS_500/build/lynxos/120604-C/x86/lib/libnsl.a(getpwent.as.o): In function `getpwent': getpwent.as.o(.text+0xb4): undefined reference to `atoi' getpwent.as.o(.text+0xce): undefined reference to `atoi' /export/build1/LYNXOS_500/build/lynxos/120604-C/x86/lib/libnsl.a(getpwent.as.o): In function `getpw': getpwent.as.o(.text+0x29e): undefined reference to `atoi' /export/build1/LYNXOS_500/build/lynxos/120604-C/x86/lib/libnsl.a(getpwent.as.o)( .text+0x34c): more undefined references to `atoi' follow collect2: ld returned 1 exit status make[4]: *** [named] Error 1
It seems that the problem is in strtol.c version 10.2, which says that the definition was moved to stdlib.h, and stdlib.h appears to have it, yet the build fails anyway...
Looking at the compilation line for strtol.c I see it produces strtol.as.o, which in a previous successful build yeilds:
$ nm strtol.as.o U __ctype U __divdi3 U __moddi3 U _errno 00000208 T atoi 000001f4 T atol 00000000 T strtol
But in this build yeilds:
$ nm strtol.as.o U __ctype U __divdi3 U __moddi3 U _errno 00000208 T atoi 000001f4 T atol 00000000 T strtol
So how do we fix this error?
The daily CVS Checkin Log works by creating a temporary CVS area based on a tag then doing a cvs update capturing what files have changed. Those files are examined and a report is produced by ECR number.
While this works fairly well and is relatively optimized (only working on the files that changed since the label) it may not be that accurate. It seems to assume that all files for an ECR are checked in together and on the same day. What if, for example, somebody checked in a file for an ECR days ago and it now checking in the rest of the work? The files reported for this ECR may be incomplete.
Here's an example of where this happened. In today's CVS Checkin Log email the following files are listed as associated with ECR # 20591:
ECR Number: 20591 src/lib/libc/strto_int.c 10.2 zhuravle 2004/12/27 16:11:21 src/lib/libc/strto_real.c 10.2 zhuravle 2004/12/27 16:09:47 src/lib/libc/strto_real.h 10.2 zhuravle 2004/12/27 16:09:47 src/lib/libc/strtod.c 10.4 zhuravle 2004/12/27 16:09:47 src/lib/libc/strtof.c 10.2 zhuravle 2004/12/27 16:09:47 src/lib/libc/strtoimax.c 10.2 zhuravle 2004/12/27 16:11:58 src/lib/libc/strtol.c 10.2 zhuravle 2004/12/27 16:11:58 src/lib/libc/strtold.c 10.2 zhuravle 2004/12/27 16:09:48 src/lib/libc/strtoll.c 10.3 zhuravle 2004/12/27 16:11:58 src/lib/libc/strtoul.c 10.2 zhuravle 2004/12/27 16:11:58 src/lib/libc/strtoull.c 10.3 zhuravle 2004/12/27 16:11:58 src/lib/libc/strtoumax.c 10.2 zhuravle 2004/12/27 16:11:58 usr/include/rcsid.h 10.2 zhuravle 2004/12/27 15:57:50 usr/include/stdlib.h 10.10 zhuravle 2004/12/27 16:07:26
Yet thoroughly scanning the CVS logs we find:
ECR Number: 20591 Nbr of files: 25 Nbr Path/File Version Author Date --- ----------------------------------- ------- ----------- ------------------- 1 strtoull.c 10.3 zhuravle 2004/12/27 16:11:58 2 strtoull.c 10.2 zhuravle 2004/10/22 12:04:48 3 Makefile 10.18 zhuravle 2004/10/22 12:04:47 4 strto_int.c 10.2 zhuravle 2004/12/27 16:11:21 5 strto_int.c 10.1 zhuravle 2004/10/22 12:04:48 6 strtold.c 10.2 zhuravle 2004/12/27 16:09:48 7 strtold.c 10.1 zhuravle 2004/10/22 12:04:48 8 strtoll.c 10.3 zhuravle 2004/12/27 16:11:58 9 strtoll.c 10.2 zhuravle 2004/10/22 12:04:48 10 strtoul.c 10.2 zhuravle 2004/12/27 16:11:58 11 strtoul.c 10.1 zhuravle 2004/10/22 12:04:48 12 strtoimax.c 10.2 zhuravle 2004/12/27 16:11:58 13 strtoimax.c 10.1 zhuravle 2004/10/22 12:04:48 14 strtoumax.c 10.2 zhuravle 2004/12/27 16:11:58 15 strtoumax.c 10.1 zhuravle 2004/10/22 12:04:48 16 strto_real.c 10.2 zhuravle 2004/12/27 16:09:47 17 strto_real.c 10.1 zhuravle 2004/10/22 12:04:48 18 strto_real.h 10.2 zhuravle 2004/12/27 16:09:47 19 strto_real.h 10.1 zhuravle 2004/10/22 12:04:48 20 strtod.c 10.4 zhuravle 2004/12/27 16:09:47 21 strtod.c 10.3 zhuravle 2004/10/22 12:04:48 22 strtof.c 10.2 zhuravle 2004/12/27 16:09:47 23 strtof.c 10.1 zhuravle 2004/10/22 12:04:48 24 strtol.c 10.2 zhuravle 2004/12/27 16:11:58 25 strtol.c 10.1 zhuravle 2004/10/22 12:04:48
Now files4ecr.pl thoroughly checks all CVS logs (based on the paths it is givin that is) but this takes a lot of time.
Note that Makefile is listed by files4ecr.pl but not in the daily CVS Checking Log. That's because Makefile was checked in with that ECR number a while ago. The daily CVS Checkin Log does not reflect this historical fact.
Also note that ECR's unlike tags that can only exist on one revision, ECR numbers are just strings placed in the check in comment. Thus there is the possibility that they will exist on more than one revision (e.g. strtod.c versions 10.4 and 10.3). IOW checking in another version of a file and using the same ECR number will not remove the ECR number in the command of a previous version of the file!