Core Minutes 4/10/2012

• ScienceTools: (Jim) New release ScienceTools-09-28-00 was tagged last week which includes all fixes since Feb. 22nd. See Science Tools Development Notes for details.

• FSSC: (Eric) The FSSC Snow Leopard binary distribution of ST does work on Lion.

• Reprocessing: (Tom G.)

  Reprocessed started up again last Friday and has been going full tilt. Currently 35% of the total have completed. The new files produced are currently 270 TB of space. 210 TB will be reclaimed by moving the old recon files. We are also down to 83 TB of remaining xroot space - that's after 4 new servers put in play a month ago. Wilko is commissioning the 5th server and hopefully it will be online later today. Then the recon file purge can start today or tomorrow.

  The problems reported last week have been mostly addressed.

  Tom is out tomorrow through Monday. Warren has kindly agreed to keep an eye on the reprocessing during that time. Tom will be online from time to time as well.

  Richard asks, how many cores are we getting these days? Tom's eye-ball estimate is that we fluctuate from 1200-1300 cores to peaks of 2500 cores. The peaks don't last long and occur about once a day. 1800 is probably the average. Richard says that's good since our allocation is 1600.

  Richard asks about the FT1 to astroserver status. Brian V is handling that. There were a number of Oracle DB issues that needed to be addressed last week. Brian is now back to loading the first year FT1 files. He encountered a new problem yesterday, where Oracle required a new account for each set of files loaded. Tom believes that issue has been solved. Once the loading starts, it should time a couple of days to complete - which would be time-early.

• IFC Meeting: (Richard) Next IFC meeting is in 2 weeks. Working to produce a budget. Waiting on an estimate from the DB guys for replacing the oracle server. For now, the budget assumes the cost will be similar to the cost to buy the current server 4 years ago. In addition there will be purchases of 500 TB disk, 1.5 PB tape, refreshing 400 cores in batch, and a handfull of login servers.

• Mac Support: (Richard) We own the only 2 Mac servers in the entire lab. The Computing team is wondering if they really want to expend the man-power on support. We were asked to assess our true need for Mac. The FSSC was queried and they were not thrilled at the prospect. In addition, the FSSC does not distribute the full SciTools. The LAT provides the GRB and Pointlike tools.

  If the Computing Center holds firm, we'll have to find a way to do without them. We've asked them to quantatively estimate how much effort they truly expend on the Macs. In addtion, the servers are getting old (~4 years) and should be replaced. However, Apple is out of the server game, so if we do replace them, we'll need to purchase a desktop machine. Perhaps the desktop guys could provide support if that is the case. Tom has also suggested we may be able to avoid the use of LSF and use cron instead. This is something we'll likely have to deal with this year. Snow Leopard will likely fall out of favor by the Fall when Apply likely stops issuing patches, and it is decided that Snow Leopard is a security risk.

• MySQL migration: (Heather) No real change from last week: everything is ready to go except for OpsLog because the responsible person (Tony J) has been out of town and unable to test it. We've been reminded that we need to eliminate our RHEL4 boxes ASAP, which includes glastlnx01, lnx02, and lnx14. Hence, we need to complete the GR migration to RHEL5 pronto.

• Pass7 news: Leon has been busy sneaking P7 fixes into the P8 version of the code. Thursday will mark the start of one of two truncation orbits.

  There is an issue with the increased data volume which may require a fix to the onboard data handling and Richard wonders if that has been handled yet? Leon responded that it has not been handled. The data increase is 10% and is considered within acceptable limits if there is advanced planning for extra downlinks. Richard wonders if there is an estimate on a real fix? JJ estimates such a fix will not take weeks or years.

• Pass8 news: (Tracy) Tracy's efforts are focused on GoGui and Win 7.

• Core talk:

  • Systests (Heather) Heather has continued to run system tests on the new L1proc GR 17-35-24-lp22 as well as the new P8 v19r4p1gr15. When attempting to run the P8 system tests, we were remined that the JO files for running L1proc GR are different than the P8 JOs - in particular the G4 physics list is different. There was also some discussion in February concerning trouble with the AllGammaOverlay test, which Leon suggested could be fixed with some new Tracy cut that filters the "problematic" events. It seems that JO parameter is not yet in the system test JO and probably should be added.

  • New SCons version (Heather) We completed the move to SCons 2.1.0 for the RM last week. Tom installed the newer SCons on glast-win04 and set up the windows builds to use it. No trouble so far. We should probably alert the user community as well as update the workbook.

  • First Tagging Failure (Tom) We had our first tagging failure after Tom implemented the changes for error handling in RM. The email notification worked perfectly. Unfortunately, the build did not terminate as he expected. Tom has added more logging information and will hopefully gain more insight with the next failure. For the record, this error was due to a collison between tagCollector running on GR and RM tagging LATEST on ST. The tip package was being accessed by both activities.

  • GR vc90 (Tracy) Tracy has gotten GoGui going on his laptop as well as the HEAD of GR. He now has a new CLHEP 2.1.2.2 build provided by Heather which was built via CMAKE rather than cygwin. This cleared up the issue Joanne reported a few weeks back concerning the CLHEP Matrix classes. Heather has also rebuilt G4 to use this newer CLHEP and will be providing Tracy the binaries later today. Meanwhile, Tracy has pushed ahead and has built and run TkrRecon's unit test. Tracy is still unable to run Gleam due to an inability to load up the CAL and ACD. Tracy is looking forward to Joanne's quick return. Meanwhile, he also has some thoughts about the new systems and doesn't particularly enjoy

    1. The copy of the source code that SCons generates, requiring one to modfiy that copy to do a rebuild, that is distinct from the CVS original.
    2. The new location of the runtime jobOptions files, which is located at the top-level of the GlastRelease and not within the packages themselves.
    3. Issues with GoGui and Visual Studio putting their build products in different locations, which causes VS to think libraries need to be rebuilt even though one has already built them with GoGui.

    Heather started up a Confluence page to capture user comments concerning SCons and GoGui: https://confluence.slac.stanford.edu/display/SAS/Comments+from+Fermi+SCons+Users

previous

minutes index

J. Bogart, Last Modified: