Design Thoughts as of January '06

Db design

The MOOT db is going to have an uncomfortably large number of tables which can be classified in various ways. One is by function:

Database multiplicity

However, this doesn't tell the whole story because some of these (the class tables and relation tables relating two classes) are read-only for user applications: only the MOOT maintainer needs to write to them. For the others, user applications at most need to write to them for a particular instrument.

In order to minimize dependence on network access, I expect to create separate MySQL databases for each site; e.g. mood_SLAC, mood_NRL, mood_MOR. The reasonably static class and class-to-class tables would appear in each database, of identical form and normally of identical contents. The other tables (e.g. instance tables like Parameters) would have the same form from one database to another but contents would be entirely independent. Each site would have a copy of "its own" database which would be writable by users with sufficient privilege. It might also optionally have read-only mirror copies of one or more of the other databases.

Might there be a site which is the primary site for more than one instrument? If so, either will have to add an 'instrument' column to various instance and (maybe) relation tables, or will need to multiply the number of such tables by the number of instruments. Or one could have still more databases: one per instrument rather than one per site. Ugh.

Services

This list is probably not complete.

Connect

Arguments would include instrument, perhaps indication of whether write access is required, and something which resolves to connection parameters such as userid and host. If write access is desired, that plus instrument ought to be sufficient information to determine host.

Enter Calib Set

Input consists of 3 lists of file descriptors, one each for ancillary, Parameters, and Offline calibs. Characterization values (e.g. energy) to be associated with the Calib Set should also be supplied.

For some offline calibs (those corresponding to configuration - e.g., register settings - rather than channel response), vstart and vend should both be set to EOT ("end of time", which is Dec. 31, 2037) when they are entered into the database.

Also need to specify instrument, but this needs to be specified all the way through. Most likely it will have been specified when connecting and won't need to be respecified.

Then there is the question of the relation tables. Would be very tedious to have to specify exactly what depends on what. Perhaps it can be done with class-to-class relation tables.

MOOT doesn't do anything other than add entries to various tables for this service. There may be - probably will be - separate services for delegates like CAL with a well-defined set of characterization values. There also has to be some sort of general-purpose thing which will handle whatever is left over.

Create logical (as opposed to uploadable) Config

There may be more than one way to do this:

1. Simple-minded way: just specify a long list of Parameter entries and maybe some attributes for the Config (e.g., temperature, threshold in physics units,...) MOOT service will
   • make sure they all exist in dbs
   • form corresponding collection of FSW_input files if they don't already exist and enter them in FSW_inputs table
   • make entries in FSW_to_Parameters relation table as needed
   • make binaries from FSW_inputs source files; register with fmx; update state in FSW_inputs table
   • make a new Config entry
   • make entries in Config_to_FSW relation table.
2. By criteria. Caller characterizes desired Calib Set sources.
   • MOOT looks for most recent Calib Sets (different ones for some of the delegates) fitting the characterization. For other miscellaneous Parameter files which don't belong to a Calib Set, just pick the most recent valid one.
   • now proceed as in a.
   In either case, key for new Config is returned to caller.

Create uploadable Config

Input is Config key, from #2, and specification of destination of upload (e.g., instrument + (SIB or RAM) )

• MOOT finds collection of FSW_inputs associated with the Config
• Check whether uploadable form of each FSW_input already exists. If not, invoke fmx function to make it; update status fields of FSW_inputs rows.
• See if LATC master already exists for this collection. If not, create it, register with fmx, make entry in FSW_inputs table, etc.
• return status of the whole thing.

Query for upload

Input is Config key and destination. MOOT will

• check that all required FSW_inputs exist in uploadable form
• return list, or, better, two lists:
  • those believed to be already uploaded
  • those which definitely need to be uploaded

Note these lists do not include executable code binaries.

Confirm upload

Input is list of newly-uploaded binaries. MOOT will invoke fmx commit (or equivalent in the new fmx) for each.

Query for start-run

Input is Config key and destination. MOOT will

• check that relevant files have been uploaded
• return list of configuration files that need to appear in start-run.

Log configuration change

Something (FASTCOPY process?) will invoke this service when a new configuration record is detected in the data stream.

If the configuration has not changed, MOOT will at most update a timestamp field in the most recent row in the History table.

If it has changed, MOOT will

• check whether or not the configuration record is associated with data taken after any data seen so far, or whether it comes from an old part of the data stream, delivered to the ground out of order. These two cases will have to be handled somewhat differently in the steps to follow.
• add a new row to history table and maybe update a field in the old current row
• trace back information in config record, which identifies certain FSW inputs, as follows:
  • references in configuration record are of two types: direct (to some FSW_input like JJped or LCI script) or indirect (to file-of-files like LATC master). In latter case, resolve to files being referred to by file-of-files. Final result will be a collection of FSW_input entries.
  • For each FSW_input, use relation table to find related Parameter file entries. If there is more than one Parameter file entry of a particular class (and probably also will need to match tower number) which is related to a FSW_input, take the most recent.
  • Similarly, work back from Parameter file entries to Ancillary file entries.
  • For some kinds of offline calibration (the ones that correspond directly to configuration changes such as settings for threshold registers), if vstart > config change timestamp, set vstart to this timestamp. Also look for previously-active calibration of this type and set its vend.
• Maybe also maintain a History_to_FSW relation table (or other relation table linking History to one or more of the file collections). The first step above to discover FSW_inputs from configuration record may be ugly enough that it would be well to do it only once.

Find "correct" calibration

Given a timestamp, calibration type, and other criteria (e.g., flavor), find the right calibration entry. This is already done as part of the old offline calibration system. It will need small mods when the calibration metadata table moves to the MOOT dbs.

Find "correct" configuration

Given timestamp and config file class (FSW_input or Parameter; for most applications Parameter will be more interesting) find the instance, if any, in effect at specified time. (If Parameter file, may also have to specify tower, or will separation by tower only be allowed for Ancillary files?)

• find correct history entry, work back to FSW_inputs or use information stored in History_to_FSW relation table.
• if Parameter requested, work back another level to Parameters in use.

Unlike offline calibrations, there is no question of flavor or other criteria here. Only one configuration file of a particular class (and tower, if applicable) can be active.

Supersede a calibration

Suppose someone detects a bug in the procedure which generates an offline calibration. One might want to regenerate that kind of calibration with a new algorithm from the original input. In that case the new calibration should be accessed for reprocessing. The existing supersede function for offline calibrations does most of the right things, but for MOOT one would also have to update any relation tables involving offline calibrations.