| LAT Communication Board Driver: Software Architecture and Interfaces | ||
|---|---|---|
| Prev | Chapter 2. Interrupt Mode Driver | Next |
This section describes the sending of commands and the processing of results with the LCBD. Commands are bit sequences sent down to the front end electronics (register reads, register writes and dataless commands ). Every command generates a result – register writes and dataless commands generate a simple "command successfully transmitted" result, while register read commands return the register value as a result.
Some definitions used within this section:
A single command sent to the front end electronics.
An ordered list of command items containing one or more command items.
A single result from the LCB in reply to a command item.
An ordered list of result items from the LCB in reply to a command list.
An indivisible unit of work consisting of a command list and its associated result list.
The LCB has the capability of bundling several command items together and sending them sequentially as a command list. After all the commands are processed and all the responses are received the LCB fires an interrupt that the driver traps. The driver then notifies the user – see Section 2.3 for more details.
This arrangement allows for asynchronous commanding, e.g. a user can queue a command list to the LCB, continue to do other work and then be notified asynchronously that the results are ready for processing.
For the case when a user only wants to send a single command it is purposed to also offer a synchronous interface to the LCBD. With this interface a user would queue a single command to the LCB and wait for the command to complete. At this time it appears straight forward to build the synchronous interface on top of the asynchronous interface.
The asynchronous interface offers a powerful way for users to queue command lists to the LCB for processing. While the LCB is busy processing the command list and receiving result items the user can continue to do other work. The LCBD notifies the user after the final command item and result item are processed (see Section 2.3). After notification the user would proceed to inspect the result list.
The LCBD provides a data structure for organizing the command list and the result list called the LAT I/O Transaction (LIOX). Users communicate with the LCBD via an opaque handle to a LIOX structure. A diagram showing the relationship between the command list, result list and LIOX structure is shown below in Figure 2-4.
In addition to organizing memory the LIOX also maintains an internal command pointer used when filling the command list. The command pointer behaves much like a file pointer – it always points to the memory location where the next command will be inserted. As commands are added to the command list the command pointer is updated to point to the next available command slot.
The LIOX also performs bounds checking, i.e. it will not allow more commands to be added than will fit into the available memory for the command list.
Before creating and sending command lists via the LCBD a LIOX handle must first be properly initialized. This entails the following activities:
Allocate memory for the LIOX structure, the command list and the result list.
Implement a user-defined result processing function called during result dispatch (see Section 2.3).
Initialize the LIOX by passing in pointers for the command list memory, for the result list memory and for the result processing function.
Once created a LIOX handle can be in one of the following states:
| LIOX State | Description |
|---|---|
| LIOX_READY | Allocated, ready for transmission |
| LIOX_PEND | Command list sent, waiting for result |
| LIOX_RECEIVED | Result received |
Table 2-1. LIOX Handle States
The allowable state transitions are shown below in Figure 2-5. These states are discussed in the following sections.
Memory allocation is left entirely to the user, i.e. the LCBD never allocates memory of its own. Memory must be allocated for the LIOX handle, the command list and the result list.
The LCB places constraints on the memory allocated for command lists and result lists. In particular the memory for the lists must be visible to the PCI bus and have the size and alignment attributes shown in Table 2-2.
| Memory Space | LCB DMA | Size | Alignment |
|---|---|---|---|
| LIOX Handle | N/A | < 100 Bytes, fixed. | None |
| Command List | Read | Up to 4092 Bytes | 512 Byte |
| Result List | Write | Up to 4084 Bytes | 8 Byte |
Table 2-2. LIOX List Memory Attributes
All of the above objects could be pre-allocated to form object pools. The user could request an object from the object pool as needed.
The number of allocated lists is a function of the number of :
Concurrently executing command lists, e.g. background housekeeping during other commanding.
Command lists previously prepared, e.g. command lists containing default/baseline configuration commands.
It will be worthwhile for the LAT system software to offer memory allocators for different alignment requirements. These allocators could be part of the LAT FSW utility library.
After a result is received and the result is processed the user can return the objects to the appropriate object pool. Objects can be safely deallocated according to the following table:
| Object | LIOX States OK to Deallocate |
|---|---|
| LIOX Handle | LIOX_READY, LIOX_RECEIVED |
| Command List | LIOX_READY, LIOX_RECEIVED |
| Result List | LIOX_READY, LIOX_RECEIVED |
Table 2-3. Deallocating Memory
It is unsafe to deallocate memory when the LIOX state is LIOX_PEND. The LCB will DMA the result into the result list whenever the result arrives. Once a LIOX is initiated the LCB "owns" the result list and command list memory until the transaction completes.
This means we cannot safely deallocate the memory for the result list or the command list until the transaction completes – if the transaction never completes we lose the memory associated with both lists and the memory for the LIOX handle.
Initializing a LIOX handle requires the following steps:
Allocate memory for the LIOX handle.
Allocate memory for the command list.
Declare the response handling function.
Allocate memory for the result list (optional).
Allocating memory for the result list may be deferred until the command list is completely loaded, at which time the total amount of memory required for the result list is known. If, however, the LIOX will be used for synchronous operations ( see Section 2.4.2) the result memory must be supplied at initialization time.
The function prototypes for the response handling function and the initialization function appear below:
int LIOX_Init(LIOXHandle lh, unsigned short cmdLen, unsigned int *cmdList, unsigned short rspLen, unsigned int *rspList, LCB_resultFunc func, void *usrData);
The following pseudo-code illustrates the initialization of a LIOX Handle.
// declare result processing callback and extra data
static int myProcessResultFunc( LIOX lh, void *userData);
unsigned int myExtraData;
// allocate memory for command list and LIOX struct
void *pCmdList; // command list
void *pLIOX; // LIOX handle
unsigned short cmdLen; // command list length
LIOXHandle lh;
// allocate memory for the command list -- suitably aligned
cmdLen = 4080;
pCmdList = mallocCommandList( cmdLen);
// allocate memory for the LIOX structure
pLIOX = mallocLIOX( LIOX_SizeOfIOX());
// create opaque handle to LIOX -- note the result list length
// is zero and the result list pointer is NULL. These will be
// provided later when the command list is queued.
lh = LIOX_Init( pLIOX,
cmdLen, pCmdList,
0, NULL,
myProcessResultFunc,
myExtraData);
// LIOXHandle lh is now initialized and ready for use
|
The allocation of the result list memory is deferred until after the command list is completely filled. At that time the required size of the result list is completely determined and the exact amount of memory can be allocated. See Section 2.4.1.3.
After the LIOX handle is initialized the internal command pointer points to the first available command slot in the command list. A mechanism exists for reseting the command pointer so that a LIOX handle can be re-filled with a new list of commands.
Commands are added to the command list by passing the associated LIOX handle as an argument to the family of "LAT Command Functions". The "LAT Command Functions" provide a basic interface for read/write and dataless commands to the various addressable objects within the LAT.
As of this writing 11 types of addressable objects exist within the LAT – this number is expected to grow to about 20 types of objects. Each object contains registers which are also addressable. The following object types exist today:
TEM Common Controller
TEM Trigger Interface Controller (GTIC)
Tracker Cable Controller (GTCC)
Tracker Readout Controller (GTRC)
Tracker Front End Controller (GTFE)
Calorimeter Cable Controller (GCCC)
Calorimeter Readout Controller (GCRC)
Calorimeter Front End Controller (GCFE)
AEM Common Controller
ACD Readout Controller (GARC)
ACD Front End Controller (GAFE)
For each object type there exists three "Command Functions" – register load, register read and dataless command. That is a total of 33 functions for 11 object types. A rough estimate projects 50-60 functions ( about 20 object types ) in the final system.
In addition to the above command/response items are three special command items:
Command Fabric Reset
Injected Time Markers
Sending bulk data on the event fabric
The following illustrates the queueing of a normal command item.
Example 2-1. Reading A Calorimeter DAC
As an example consider the hypothetical case of reading a DAC on a calorimeter front end chip (GCFE). The reading of a DAC register corresponds to a register read "Command Function". To uniquely address a DAC register on a specific GCFE requires a TEM address, a GCCC address, a GCRC address, a GCFE address and the DAC register number. The GCFE read "Command Function" requires all these parameters as arguments. Below is the function prototype for LIOX_qGCFEload.
int LIOX_qGCFEread(LIOXHandle lh, unsigned short temId, unsigned short gcccId, unsigned short gcrcId, unsigned short gcfeId, unsigned short regId);
Calling LIOX_qGCFEread with appropriate arguments would add a GCFE read command to the command list of the LIOX handle. The LIOX handle would update its internal command pointer to point at the next available command slot in the list. An example is shown below:
unsigned int errVal; LIOXHandle lh; /* previously initialized */ /* queue the read command to the command list */ errVal = LIOX_qGCFEread( lh, temId, gcccId, gcrcId, gcfeId, DACregId); |
This process is repeated, adding commands to the LIOX handle until the entire command sequence is loaded or the LIOX handle runs out of memory. Next the command list is queued to the LCB for execution.
A special command item exists that instructs the LCB to assert the hardware reset line of the command fabric. This resets all the nodes of the command fabric. This command generates a result item that will have the reset field of the error word set.
The fabric reset command must be the only command in the command list – queueing the fabric reset command will erase any previously entered command items in the command list. This is because the fabric reset command is implemented as a null command list of zero length.
Quoting [huffer1]:
An application may need to specify an arbitrary amount of idle time between two export items, or to inject an accurate time marker in the RESULTS FIFO. Both of these requirements must be satisfied without actually transmitting a packet on a fabric ... A result is generated at a time determined by the stall/timeout field.
Queueing an injected marker command item takes the following form:
The stall/timeout is in units of sysclks, i.e. 50 nano-second increments.
Unlike the fabric reset, multiple markers may be queued per command list.
This command generates a result item when completed.
While the sending of bulk data occurs on the event fabric, the mechanism for queueing a bulk transfer looks like queueing a command item. Queueing a bulk LCB-to-LCB data transfer takes the following form:
int LIOX_qBulk(LIOXHandle lh, unsigned short nodeId, unsigned short proto, unsigned int nBytes, unsigned int *data);
For LCB-to-LCB the MSB of the 6-bit nodeId must be set, i.e. all LCBs have a nodeId with the high-order bit set. LIOX_qBulk() will fail if the nodeId is not of the proper format.
The proto parameter indicates which of the 3 LATp protocols to use.
Multiple bulk transfers may be queued in the same command list.
This command generates a result item when completed.
Once the command list of a LIOX handle is loaded the list is ready for execution. At this time the amount of result list memory required is completely determined by the command items. The result list memory is now allocated and assigned to the LIOX handle.
The LIOX_resultSize() function returns the number of bytes required for the result list. The result list memory must be suitably aligned as described in Table 2-2. The result list memory is added and the command list queued with a single call to LIOX_qCmdList().
int LIOX_resultSize(LIOXHandle lh);
int LIOX_qCmdList(LIOXHandle lh, unsigned short nBytes, unsigned int *rstList);
Behind the scenes, however, the LCBD performs various bookkeeping operations when queueing the command list. Refer to [huffer1] for more details on queueing command lists (referred to as export lists).
The address of the command list and the length of the command list are queued to the LCB by the LCBD. Together the length and address are called the export descriptor. The length of the list is calculated from the internal command pointer.
Once queued the state of the LIOX transitions from LIOX_READY to LIOX_PEND. Refer to Figure 2-5.
After queueing the export descriptor the LCB takes over. The LCB DMAs the command list from the SBC's memory to the LCB's memory and sends the commands out on the command/response fabric of the LAT one by one.
As the results come into the LCB it stores the result items locally. After the final result comes in the LCB DMAs all the result items to the result list associated with the command list that initiated the commanding sequence. It next puts a result descriptor in the result FIFO and fires an interrupt (if the FIFO transitions from empty to non-empty).
The LCBD traps the interrupt and reads the result descriptor from the results FIFO. The LCBD passes the result descriptor into the result dispatch unit (see Section 2.3) for routing to the appropriate message queue.
Once a result list is ready the LCBD queues a LIOX handle to the Solicited Response Queue (see Figure 2-1). Ultimately this results in a callback to the user supplied solicited response handler.
Using the LIOX handle the user navigates and processes the result list. When processing is complete the user may free the LIOX handle and the associated command list and response list.
The LCB provides two types of result items, both of which are fixed in length. The first type of result item is a simple "transmission verification" result used for command items, which by their nature have no response data. This result type contains a transmission timestamp and error information.
The second type of result item is in response to a read command. In addition to the "transmission verification" data of the simple result type, the response type result item also has a payload containing protocol header information and the response returned from the target.
The next sections describe methods provided by the LCBD for navigating the results list and for decoding result items.
This section assumes that a result list has been successfully dispatched to the user application. Section 2.3 describes how results are dispatched.
The LCBD provides a basic mechanism for walking or iterating over the result list, item by item. The LIOX handle maintains a result pointer (see Figure 2-4) that points to the next available result item.
The LCBD provides the following functions for result list navigation:
The LIOX_nextResultItem() function returns a pointer to the next ResultItem and updates the result pointer for the LIOX. When no more result items exist the functions returns NULL. The LIOX_rewindResults() resets the results pointer to the beginning of the result list.
A user can iterate over a result list as follows:
int myRspCallBack( LIOXHandle lh, void *usrData) {
int status = LCB_OK;
ResultItem *pItem = NULL;
// process result items
while ( (pItem = LIOX_nextResultItem( lh))) {
status = processItem( pItem, usrData);
}
// At this point it is the user's choice to free the LIOXHandle
// and associated command list memory and result list memory.
// You may want to keep it around for later execution.
return status;
}
|
In the above example the while loop terminates when LIOX_nextResultItem returns NULL.
As alluded to earlier two types of result items exist. The two types have some data fields in common while the "response" result type has more information and requires special decoding.
This section discusses functions used to access the common data fields for both result types.
Both result types have a timestamp field (24 bits), an error field (16 bits) and a result type bit. The following functions retrieve these fields from a ResultItem pointer.
unsigned intLIOX_riGetTimestamp(ResultItem *pItem);
unsigned intLIOX_riGetError(ResultItem *pItem);
intLIOX_riHasPayload(ResultItem *pItem);
The LIOX_riHasPayload() function test the result type bit and returns non-zero if that bit is set. This indicates that the result item is a response to a read command and has a payload.
Result items that contain payloads in response to read commands require extra decoding. All response result items contain two fields: a LATp Header and a data payload.
The LATp header is a 16 bit field, while the data payload is a 112 bit field ( 7 16-bit integers).
The following functions are available for decoding response result items:
unsigned shortLIOX_riGetHeader(ResultItem *pItem);
unsigned short*LIOX_riGetPayload(ResultItem *pItem);
In addition to the basic LIOX_riGetPayload() function the following helper functions might also be added for specific payload types.
unsigned shortLIOX_riGetPayload16(ResultItem *pItem);
unsigned intLIOX_riGetPayload32(ResultItem *pItem);
unsigned long longLIOX_riGetPayloadTKR64(ResultItem *pItem);
The LIOX_riGetPayload16() and LIOX_riGetPayload32() return the first 16 bits and 32 bits of the payload respectively. Most register reads fall into these two cases.
The LIOX_riGetPayloadTKR64() function is a special decoder for tracker register reads (GTRC and GTFE registers are 64 bits wide). The raw data from a tracker register read contains extra protocol bits every 16 bits – this function removes the protocol bits and returns only the 64 bit register value.
Other special decoders will be needed to facilitate decoding of special registers. The environmental registers of the GTIC is one example.
Example 2-2. Reading a Calorimeter DAC – Continued
Continuing the example started in Example 2-1 we will decode the 16 bit GCFE DAC value within our result list callback function. Assume the result list only contains a single result item, the result item that corresponds to the LIOX_qGCFEread() command.
int myRspCallBack( LIOXHandle lh, void *usrData) {
int status = LCB_OK;
ResultItem *pItem = NULL;
unsigned int timeStamp;
unsigned int errorValue;
unsigned short int registerValue;
/* fetch the first result item from the LIOX handle */
pItem = LIOX_nextResultItem( lh);
if ( pItem != NULL) {
// get the time stamp value
timeStamp = LIOX_riGetTimestamp( pItem);
// get the error value
error = LIOX_riGetError( pItem);
// test if this item has a result payload -- in this contrived
// example we already know it does have a payload.
if ( LIOX_riHasPayload( pItem)) {
// By design this item is a 16 bit GCFE register
registerValue = LIOX_riGetPayload16( pItem);
// do something useful with register value
}
}
return status;
}
|
The synchronous interface is used when the user wants to send a single command and is willing to block waiting for the result to come back. This interface could provide backward compatibility with the GNAT interface that the I&T test stands are currently using.
The synchronous interface would be built on top of the asynchronous interface, hiding the details of the asynchronous interface from the user.
The synchronous interface is much simpler than the asynchronous interface. With the synchronous interface the user only needs to call the synchronous version of a "LAT Command Function". However, the user must still allocate memory for the command and result lists and properly initialize a LIOX structure.
The synchronous versions of the "LAT Command Functions" have similar function names and signatures as the asynchronous versions. The synchronous function name uses the letter s where the asynchronous function name has the letter q.
Below is the function prototype for the synchronous version of the GCFE read register command discussed previously in Example 2-1. This function reads a register on a particular GCFE.
int LIOX_sGCFEread(LIOXHandle lh, short temId, short gcccId, short gcrcId, short gcfeId, short regId, short* value);
Example 2-3. Read Calorimeter DAC – Synchronous Interface
With the synchronous interface the entire process of sending a command, waiting for a result and decoding a result is reduced to a single synchronous function call. Reading a calorimeter register is implemented as follows:
unsigned int errVal;
unsigned short int regVal;
/* read the register value */
errVal = LIOX_sGCFEread( lh, temId, gcccId, gcrcId,
gcfeId, DACregId, ®Val);
/* do something useful with register value */
|