Core Minutes 2/10/2015

• ScienceTools: (Jim) See Science Tools Development Notes for details of these and several other updates.

• FSSC: (Joe? Alex?)

• Weekend debacle: (Tom et. al.) Starting around 2 AM on Sunday many jobs failed because they could not connect to the MySQL calibrations metadata database. The reason was too many connections. The MySQL server connection limit is set to 800 and it was being exceeded. This is not normally an issue since connections are released a few events into a job, but on Sunday many jobs were holding on to connections for hours. Ultimately, by looking at a log of such a job, it was determined that an entirely different part of the code, having nothing to do with MySQL, was throwing an exception, and the handling was such that the job would hang. The exception was being thrown from within PtValsAlg because the event timestamp was past the time it could compute a magnetic field solution (namely start of 2015) without extrapolation. A few weeks ago Jim had made a new version of the file in the astro which throws the exception, to allow the computation to proceed for times before 2020, but this new version was not in use by the reconstruction code. For now in these recon jobs Warren has set the environment variable IGNORE_IGRF_BOUNDARY, causing the code to continue without exception no matter what the year is, thus eliminating the hangs and MySQL connection issues.

  To be explored:

  • Decide if, in the short term, we want a release branched off of GR 20-09-02 (now in use for recon) which has the newer astro tag.
  • Decide what strategy we should be following with respect to allowing or disallowing the magnetic field calculation.
  • Figure out why the error was causing jobs to hang rather than crash. If they had just crashed, the source of the problem would have been found much sooner.
  • Implement better monitoring of the state of the MySQL server. It wasn't the root of the problem this time, but if we had known earlier on that jobs were staying connected for hours rather than minutes and that we were approaching the max connections limit it would have helped.

• C & A: (Leon) A couple issues from the last C & A meeting:

  Comparisons: 20-09-02 versus 20-10-xx; 32-bit versus 64-bit.

  Event-by-event comparisons always fail in the sense that there is some difference, either due to round-off or, sometimes, a larger-scale difference triggered by round-off errors. This is what happened the last time we had to migrate to a new platform. At some point we said "good enough", and that is what what will eventually happen this time, but we're not there yet.

  Not-so-isotropic background at low energies.

  Angular distance to the earth's limb is a big factor. The variable zmax represents the polar angle of the gamma with respect to local zenith. When this angle gets too large, the chances increase that the gamma is actually coming from the limb, especially at low energy where the direction of the gamma is poorly measured. Changing the cut on zmax from 100 degrees to 80 cleans things up considerably with no apparent ill effects.

  Meanwhile, Michael has been doing some fine work profiling GR using tools built into Gaudi. See JIRA LPATE-181 for details; it's quite illuminating.

• Core talk:

  • ScienceTools and Anaconda python: JIRA STGEN-148 Anaconda python was installed in GLAST_EXT for the linuxes and new test build of ST created. The FSSC requests no upgrade until the regression testing has been updated to test python changes. It is unclear who will be working on that. Jim attempted to perform an FSSC ST install at SLAC but was unable to due to the need to be "root" to run "make" for some of the packages. Joe has a workaround, and plans to send it to Jim. (as of 2/17/2015 that hasn't happened yet) Jim also asked that in the futre we try to keep ST updated with the latest cfitsio, to avoid migration issues by waiting too long. We have both signed up for the heasoft mailing list, to be alerted about new releases.

  • Other externals (Heather) cfitsio 3.37 and ape 2.8 were built in GLAST_EXT and a new SciTools build was started. As it turns out, tip's unit test failed, as did the other packages that use cfitio. There was also at least one failure associated with ape. The FSSC has been alerted and will work to find the problems. In the meantime, GGR and ST will not upgrade and it is unclear when our next opportunity will arise.

previous

minutes index

next

H. Kelly, Last Modified: