Core Minutes 4 June 2013

Core Minutes 6/4/2013

Present: Joe Asercion, Joanne Bogart, Jim Chiang, Richard Dubois, Warren Focke, Tom Glanzman, Heather Kelly, David Landriu, Leon Rochester, Tom Stephens, Tracy Usher

ScienceTools: (Jim)
- There is a new tag of caldb including P7REP v15 irfs.
- A new tag of pyLikelihood fixes a problem reported by Jean.
- There is a new tag of ST, 09-32-00, incorporating updats since March 29.
See Science Tools Development Notes for details.
FSSC: (Joe) No news.
Pass7 Reprocessing: (Tom G.) Re-reprocessing of FITS files will likely start today, should be completed in one or two weeks. He will also do a backfill. The public release will happen around July.
Pass8 (Tracy) 20-08-14 was the first attempt at a final candidate. Tom G. ran his tests. Memory leaks have been plugged (though memory usage is still high). The output root file was unpleasantly large. cpu time was well behaved (averaging about a second per event, tails generally within 10 seconds) except for a very occasional event taking 15 minutes. Measures have been taken to address this in the next release. For the rest, energy recon accounts for the largest fraction of time used (40%) but not much can be done about it; Philippe has shown that we can't speed it up without losing resolution. The next biggest component is cosmic ray tracking. An initial look at file sizes produced by Tom's test was disappointing, but the test had produced the full file rather than the compressed one. In a comparison of one compressed file to Pass7 output, the Pass8 file was about 1.2 or 1.3 times the size for Pass7. Declaring success!

The new GR, including the fix for the 15-minute events and updates to reduce ACD output size, will be made soon, hopefully today. Need to redo merit comparison. MC datasets must be generated for validation; could take a week or two.

(Richard) What about merit? Do we need to trim number of variables? (Tracy) Plan is to start reprocessing with full merit output. Later we will go to the new reduced set, but even full merit is not large enough to make much difference.
Hardware: (Richard) All the bits of the new 340 Tbyte fileserver have arrived and are being installed. It's higher density than anything we've used previously.

We are now able to order the Oracle servers.
Core talk:
- Computing meeting (Heather) We met with Shirley, Yemi and others to discuss a variety of issues:
  - Migration to rhel6 They would like to ditch rhel5 as soon as possible. We will follow a path similar to the one for the rhel4-to-rhel5 migration. The first step, expected to take about a month, will be to find out who needs continuing access to rhel5 during the transition and to set up a few rhel5 machines accordingly to be used for builds, debugging, etc.
  - lsf9 They would like to move to it immediately. Although lsf9 does support Windows, we would be the only user; they would rather not support it. We'll continue work on getting Windows builds going with Jenkins (see below).
  - Power outage (Richard) Our key resources were available during the outage on back-up power, but the wain36 server did not survive and is still down. It has data which would be accessed by jobs doing skims over the entire mission, so any such job will fail for the time being. The server may be brought back to life today. If not, we need a plan B to get the data off of it and stored elsewhere.
  - VM usage(Richard) The VMs on the new fermilnx machines are still under-subscribed. (Tom G.) Tony W and Wilko are making use of them for mysql server and xroot redirector, respectively, but they are the only ones. (Richard) Some VMs did not come back up automatically after the outage; people would like to understand why. And there are still issues concerning network.
    (Tom G.) When a machine has multiple network inteface connections one way to run them (usually considered the most desirable way) is to aggregate them, effectively making, say, two 1 Gbps links appear as a single 2 Gbps link. It does not work reliably on the fermilnx boxes. Redhat has acknowledged this as their problem and Yemi is working through the process of getting a fix in hand.
- L1 (Heather) would like to make a tag incorporating new versions of ST pacakges.
- Release Manager
  - New ST tag (Heather) No log files were visible for the rhel5 builds. (Tom S.) will investigate. [(Heather) Update: two of three rhel5-32 machines did not come back up after the outage and the third one (the slowest) timed out on the ST build. She has resubmitted.]
  - Tar balls (Tom S.) Starting at the beginning of April RM stopped creating tar balls correctly, even though there had been no change in the relevant python code. Upon further investigation he discovered the tar command being issued was strangely mangled, with spaces interpolated between every originally-adjacent pair of characters. The cause appears to be an uninitialized variable — used to keep track of tar options, it was appended-to but not initialized. That's now fixed, but we still don't understand why it stopped working when it did.
  - Windows, Mac and Jenkins (Tom S.) asked Tony to try an ST build now that the assert pop-up fix is in place (has not been implemented yet for GR). The Mac has a different problem - someone has to log in every other day or the daemon dies [lonely?].
- obf and rhel6 (Heather, Joanne) One job (so far) is known to fail on rhel6 but not rhel5 with a traceback pointing to the obf external library. Our obf externals on Linux come direct from Tony W; we just copy over libraries and header files from an export distribution he builds. Heather gently suggested to Tony that we might need a debug build; Joanne will try putting it more baldly.

minutes index

J. Bogart, Last Modified: