README.run_logs, 2005.0120 ========================== The FSW document production scripts are written to be robust in the face of missing data. In general, they will * print a nastygram (which will go into the trscrontab(1) run logs) * do something "reasonable", if at all possible Even if one of the scripts crashes, it will not abort the entire trscrontab run. And, if most of the scripts succeed, most of the web pages will be generated. On the other hand, nobody wants an incomplete and/or error-ridden set of web pages. In order to prevent this, certain kinds of maintenance must be performed. Error tracking -------------- Aside from phone calls and other personal communication, most error reports will come, via email, to the "fsw-web" role account. Occasionally, the account sees comments from web site users (including random spam). Most of the email, however, consists of run logs from trscrontab(1). There are three production trscrontab(1) runs: suncron;1400 00 4 * * * fsw_cgi/fsw_docs.d/do_spider suncron;1400 00 20 * * * bin/nitely web doxycron suncron;1400 00 20 * * * bin/nitely web Each of these runs generates a log file, which is emailed to "fsw-web". By looking over the log files, it is generally possible to determine (a) whether a problem has occurred, (b) which program is complaining, and (c) the nature of the problem. Note: the "suncron;1400" entry indicates that the cron job should be run on a "sun" (i.e., SPARC Solaris) system and that it should be allocated 1400 minutes (almost a day) of AFS write privileges. This should be enough (:-). fsw_cgi/fsw_docs.d/do_spider ---------------------------- This run traverses (i.e., "spiders") the FSW web site, including manually- edited and machine-generated web pages and the PDF versions of "Traveler" documents. The run produces assorted nastygrams, as: RobotRules: Unexpected line: Allow: ... Ignore this. The SCS folks think that "Allow" is a legal statement; the author of SWISH-E does not. Sigh. Error: Bad annotation destination Error (0): PDF file is damaged - attempting to reconstruct xref table... Ignore these. They are emitted by xpdf(1) and appear to have relevance to the actual state of the PDF files being parsed. -Failed 2 Cnt: 1060 http://.../doxygen/.../interface.html 404 Not Found text/html ??? parent:http://.../tags_by_copa.shtml This indicates that SWISH-E was unable to retrieve ".../interface.html", which was referenced by ".../tags_by_copa.shtml". Kim Lo is uncertain why these Doxygen pages might be missing, so resolving this problem may be non-trivial. -Failed 3 Cnt: 26676 http://.../0c6___F.tile_id-tile_1.shtml 500 read timeout Unknown content type ??? parent:http://.../tlm_k_L_fld.shtml Ignore this. It indicates that the web server was busy (or something). -Failed 4 Cnt: 48651 http://.../00c__F.FILEID-fileId.shtml 404 Not Found text/html ??? parent:http://.../000_P.LFILUPLEPU.shtml This indicates that SWISH-E was unable to retrieve ".../00c__F.FILEID-fileId.shtml", which was referenced by ".../000_P.LFILUPLEPU.shtml". This is being produced because cat_y2h is not dealing gracefully with bad LCAT data. Resolving this problem is likely to be non-trivial. bin/nitely web -------------- This is the main production run for the FSW documentation. It collects data from file trees, databases, and web servers, then analyzes and outputs the information as an enormous set of web pages (and a few other files). The run, controlled by the make(1) utility, has multiple stages: * bin/nitely invokes "make force". * make(1) invokes "bin/run ddf_get". This retrieves the "#DDF" lines from hand-edited files (e.g., Perl scripts, YAML configuration files), expands certain compressed information, and writes the result into "a_ddf/file_sets.yml". * make(1) invokes "bin/run ddf_y2m". This reads "a_ddf/file_sets.yml" and generates "a_ddf/makevile", a subsidiary makefile. * make(1) invokes "make -f a_ddf/makevile first". This deletes some files, forcing them (eventually) to be recreated. * make(1) invokes "make -f a_ddf/makevile". This runs the production scripts, by way of "bin/run". Each script in the run will have output of the form: bin/run foo_get \ a_etc/foo.yml vvvv Wed Jan 19 20:00:25 PST 2005 vvvv ... ^^^^ Wed Jan 19 20:00:30 PST 2005 ^^^^ This tells us that: * make(1) invokes "bin/run" with the arguments "foo_get" and "a_etc/foo.yml". The first argument is the script that needs to be run. Any other orguments are names of output files. ("bin/run" uses this list to determine whether the script is newer than any its output files.) * "bin/run" prints a header line ("vvvv ..."), invokes "foo_get", then prints a trailer line ("^^^^ ...") and some blank lines. Anything printed between the header and trailer lines is diagnostic information from the script. Anything printed outside of these lines is information about the entire run. Most scripts generate lines of the form: load_yaml; path: 'a_etc/foo.yml' This is printed just _before_ the script loads "a_etc/foo.yml". If the YAML parser complains about a syntax error, this will let you know which file had the problem. Now, let's look at some "diagnostic information". Lines that begin with "? " are warnings; lines that begin with "^ " are continuations of warning lines. To find where a diagnostic is being generated, use grep(1) on the script and/or library files ("lib/fsw_{docs,lcat}.pl"), as: % grep 'sbyte=' bin/foo_get lib/fsw_docs.pl % grep 'sbyte=' bin/cat_y2h lib/fsw_{docs,lcat}.pl "bin/cat_y2h" and "bin/cat_y2t" are rather chatty programs, particularly when they see things (generally from LCAT's XML files) that they don't like. Most of these messages come from "lib/fsw_lcat.pl", which has the common code for these scripts. ? adding 16 bits of padding in field 'FILEID', ^ package 'FILE', id 'fileId'; t4=32, t5=16 This field needed some padding, lest it be misaligned (e.g., a 32-bit field in a 16-bit location). ? align (2) incorrect for type (unsigned long long) ^ pkt_id='PowerOff', def_name='tem_off_cfg', def_type='bf' This field is specified with incorrect alignment. ? r_td->{tidy}{ISIS_ScAncillary_Pkt} undefined: pkt_type='cmd' Someone referenced a typedef (e.g., a packet) that was never defined. ? @field empty in ScAttitude: cmd_fld UINT, c_id='sec' ? missing entry in @field, t1='cmd_bf:addr-cmd_fld:lo', ^ pkt_id='LMEMDUMPMEM', lvl=1, c_type='cmd_fld', c_id='lo', ^ fld_ndx=1, #field=1 The field information array is empty or has a missing entry. ? field position error ... ? packet length mismatch ... These are (generally) "cascading" errors, resulting from earlier problem(s). ? sbit=8, s_byte=0, field='AEMFRPWRST', id='free8', package='LHK' LCAT specified a bit position (8) that exceeds 7. ? unresolved USED_BY, package='ISIS', key='att/_/E.ISIS_CMD_CNT_SEL.shtml' This attribute is defined, but not (yet) used by any field instance. "bin/con_y2h" complains about oddities in Constituent and Interface usage. These may indicate problems and really should be checked out before the FSW is released. ? no constituents for Package: FMX, LCAT, SRT, VBI ? no interfaces in 'DRV:LCB:lcb_header': mcp750, mv2304, rad750, sun-gcc "bin/ltx_get" expects to find XML files that describe test runs recorded in the LTX database. It will complain if these are missing: ? lcb_pci_register.xml nonexistent in get_test_definition /afs/slac/g/glast/flight/DRV/source/LCB/V1-1-0/ptd/LTX/lcb_pci_register.xml "bin/package_y2h" attempts to "alias" "Author" and "Manager" names that it finds in the source code. It will complain if it cannot do so: ? No alias for '' in LCI, LDF, LPA, LRA, LSM, LTC. bin/nitely web doxycron ----------------------- All of the Real Work (TM) in this run is performed by small shell scripts output by the Doxygen suite (running under CMX). If errors show up here, Tony and Kim are the responsible parties.