1.2.1. PCI Configuration Space Registers

The PCI configuration space registers are used to identify and locate the LCB on the PCI bus. Once the LCB is located the configuration space registers are used to configure the PCI memory locations for the I/O Space and the Memory Space.

The PCI configuration space registers are manipulated on long word(32 bit), word(16 bit) or byte(8 bit) boundaries at byte offsets from the beginning on the configuration space. Word in/out routines are shown below:

1.2.2. PCI I/O Space Registers

The PCI I/O space registers are all 32-bit registers. These registers control various aspects of the LCB, including LATp parameters, resets, test features and event data taking.

All registers have 32-bit read/write interface functions. Some registers also have read/write functions for accessing register sub-fields.

// Read/Write of RESET bit

// Read/Write of Event Enable bit

// Read/Write of Event Path Select bit

// Read/Write of Command Path Select bit

// Read/Write of Header Parity Definition Select bit

// Read/Write of Payload Parity Definition Select bit

// Read/Write of Inhibit Pending Event bit

// Read of Export/Result FIFO Faults bits

// Read of Event Busy bit

// Read of Board ID bits

1.2.3. PCI Memory Space Registers

The LCB's PCI Memory Space provides access to the export and result FIFOs, which are used to send and receive data. These FIFOs are 32-bits wide and 1024 entries deep, holding a total of 4KB.

When sending data a 32-bit export descriptor is written to the export FIFO. This descriptor defines where the data to be sent resides.

When receiving data a 32-bit result descriptor is read from the result FIFO. This descriptor defines where the received data resides.

The use of these FIFOs is fully described in Chapter 2. The interface functions are shown below.

1.3.1. Initialization

Initialization refers to the set of required operations necessary to put the LCB into a functioning, well known state. These operations include:

• Interface Allocation

• Board Discovery

• Default Configuration

typedef struct _LCB LCB;

LCB *lcb;

lcb = (LCB *)malloc(LCB_sizeOf());

if ( LCB_boardDetect(lcb) != LCB_OK) {
   // bad things
}
else {
   // access hardware with lcb handle
}

1.3.2. Default Configuration

The hardware interface is also responsible for leaving the LCB in a default, well known state after initialization. The default state requires programming of PCI Configuration Space registers and PCI I/O Space registers.

1.4.1. Hardware Statistics

The LCBD will maintain a count of the number of interrupts received. The interface to this counter is shown below:

In addition the LCB defines four PCI/DMA errors that can occur. The LCBD maintains 16-bit counters for each of the possible error types and provides an interface for retrieving and clearing these counters. These functions are shown below.

The LCB_ErrorType parameter is an enumerated type having four possible values, one for each of the four possible error types. The possible errors are:

• DMA of export list failed

• DMA of result list failed

• DMA of event failed

• Allocation of event data failed

1.4.2. LATp I/O Statistics

LATp defines 16-bit transmitter statistics and receiver statistics for all nodes on a fabric [huffer2]. The LCBD maintains these statistics for the LCB's communications on the command/response fabric and the event fabric. Functions for retrieving and clearing these statistics are declared below.

The fabric parameter is an enumerated type having two possible values, one for each of the two fabrics the LCB supports.

2.3.1. Command/Response Data

Once it is determined that the result descriptor is for command/response the corresponding LIOX is located. For more about the LIOX see Section 2.4.1 and Figure 2-4.

The state of the received LIOX is next changed from the LIOX_PEND state to the LIOX_RECEIVED state.

Command/response data will be given a higher priority when servicing result descriptors since it is assumed that command/response data is relatively rare compared to unsolicited data. This means the solicited message queue will have a higher priority than the unsolicited queues.

An LCB_msg object containing the result descriptor and LIOX handle will be placed on the Solicited Response Queue, allowing the processing of the cmd/rsp result to occur at task level. A task reading the message queue will call back the user supplied result processing function passing the LIOX handle and any user data as arguments.

2.3.2. Unsolicited Data

The unsolicited data can take many forms, including raw event data from event builders, filtered event data from event processing units, housekeeping data and other arbitrary data. The type of the unsolicited data, however, cannot be determined by the result descriptor alone – the LCBD will dispatch all unsolicited result descriptors based on the source packet's LATp protocol field.

LATp defines 4 protocol types [huffer2]. The LATp cell header defines the protocol type as a 2-bit field. One of the protocol types is unavailable for dispatching, leaving 3 protocols for user dispatch.

Previously (see Section 2.5) the user registered call back routines for the various LATp protocol types. The LCBD queues the unsolicited message to the queue handling a specific LATp protocol type or to the default queue if no handler is registered.

It is the responsibility of the user application on the consuming end of the message queue to process the message in a timely manner and return the unsolicited data memory to the circular buffer as described in Section 2.5.

2.4.1. Asynchronous Command and Response Interface

The asynchronous interface offers a powerful way for users to queue command lists to the LCB for processing. While the LCB is busy processing the command list and receiving result items the user can continue to do other work. The LCBD notifies the user after the final command item and result item are processed (see Section 2.3). After notification the user would proceed to inspect the result list.

The LCBD provides a data structure for organizing the command list and the result list called the LAT I/O Transaction (LIOX). Users communicate with the LCBD via an opaque handle to a LIOX structure. A diagram showing the relationship between the command list, result list and LIOX structure is shown below in Figure 2-4.

In addition to organizing memory the LIOX also maintains an internal command pointer used when filling the command list. The command pointer behaves much like a file pointer – it always points to the memory location where the next command will be inserted. As commands are added to the command list the command pointer is updated to point to the next available command slot.

The LIOX also performs bounds checking, i.e. it will not allow more commands to be added than will fit into the available memory for the command list.

// declare result processing callback and extra data
static   int myProcessResultFunc( LIOX lh, void *userData);
unsigned int myExtraData;

// allocate memory for command list and LIOX struct 
void            *pCmdList; // command list
void               *pLIOX; // LIOX    handle
unsigned  short    cmdLen; // command list length
LIOXHandle             lh;

// allocate memory for the command list -- suitably aligned
cmdLen = 4080;
pCmdList = mallocCommandList( cmdLen);

// allocate memory for the LIOX structure 
pLIOX = mallocLIOX( LIOX_SizeOfIOX());

// create opaque handle to LIOX -- note the result list length
// is zero and the result list pointer is NULL.  These will be
// provided later when the command list is queued.
lh = LIOX_Init( pLIOX, 
                cmdLen, pCmdList, 
                0, NULL,
                myProcessResultFunc,
                myExtraData);

// LIOXHandle lh is now initialized and ready for use 

            

unsigned int    errVal;
LIOXHandle      lh; /* previously initialized */

/* queue the read command to the command list */
errVal = LIOX_qGCFEread( lh, temId, gcccId, gcrcId, gcfeId, DACregId);

int myRspCallBack( LIOXHandle lh, void *usrData) {

   int        status = LCB_OK;       
   ResultItem *pItem = NULL;
   
   // process result items
   while ( (pItem = LIOX_nextResultItem( lh))) {
   
       status = processItem( pItem, usrData);
   }

   // At this point it is the user's choice to free the LIOXHandle
   // and associated command list memory and result list memory.
   // You may want to keep it around for later execution.

   return status;
}

int myRspCallBack( LIOXHandle lh, void *usrData) {

  int                 status = LCB_OK;       
  ResultItem          *pItem = NULL;
  unsigned int        timeStamp;
  unsigned int        errorValue;
  unsigned short int  registerValue;

  /* fetch the first result item from the LIOX handle */
  pItem = LIOX_nextResultItem( lh);

  if ( pItem != NULL) {

    // get the time stamp value
    timeStamp = LIOX_riGetTimestamp( pItem);
  
    // get the error value 
    error = LIOX_riGetError( pItem);
  
    // test if this item has a result payload -- in this contrived
    // example we already know it does have a payload.
    if ( LIOX_riHasPayload( pItem)) {
  
      // By design this item is a 16 bit GCFE register
      registerValue = LIOX_riGetPayload16( pItem);
  
      // do something useful with register value
  
    }
  }

  return status;

}

2.4.2. Synchronous Command and Response Interface

The synchronous interface is used when the user wants to send a single command and is willing to block waiting for the result to come back. This interface could provide backward compatibility with the GNAT interface that the I&T test stands are currently using.

The synchronous interface would be built on top of the asynchronous interface, hiding the details of the asynchronous interface from the user.

The synchronous interface is much simpler than the asynchronous interface. With the synchronous interface the user only needs to call the synchronous version of a "LAT Command Function". However, the user must still allocate memory for the command and result lists and properly initialize a LIOX structure.

The synchronous versions of the "LAT Command Functions" have similar function names and signatures as the asynchronous versions. The synchronous function name uses the letter s where the asynchronous function name has the letter q.

Below is the function prototype for the synchronous version of the GCFE read register command discussed previously in Example 2-1. This function reads a register on a particular GCFE.

unsigned int        errVal;
unsigned short int  regVal;

/* read the register value */
errVal = LIOX_sGCFEread( lh, temId, gcccId, gcrcId, 
                         gcfeId, DACregId, &regVal);
  
/* do something useful with register value */

2.5.1. Registering Call Back Handlers

The LCBD provides an interface for registering a default unsolicited data handler in addition to registering handlers based on specific LATp protocol. The declaration of a call back handler is shown below:

The declaration of the registration function for the default call back handler is shown below:

The declaration of the registration function for the LATp source address call back handler is shown below:

2.5.2. Navigating Unsolicited Data

The user needs a way to navigate the unsolicited data inside of the call back handler. From the LCB_msg item the user can extract the following information about the unsolicited data:

• Data Length

• Error Status

• LATp Cell Header

• Pointer to bulk data

The LCB_msg structure is defined below:

typedef struct __LCB_msg {
   unsigned short       error;
   unsigned short      length;
   unsigned short  cellHeader;
   unsigned short   reserved1;
   unsigned   int  bulkData[];
} LCB_msg;

unsigned short   err = msg->error;
unsigned short   len = msg->length;
unsigned short    ch = msg->cellHeader;
unsigned   int *data = msg->bulkData;

Alternatively the details of the above structure could remain hidden and inlined GET methods used instead:

2.5.3. Freeing Unsolicited Data

The unsolicited data arrives in a circular buffer managed by the LCB (write pointer) and the LCBD (read pointer). Within the handler function the user controls when the unsolicited data is freed by calling the LCB_msgFree() routine shown below.

It is important that the user return the memory to the circular buffer as quickly as possible. If the user operation will "take a long time" the user is obligated to make a copy of the unsolicited data and return the unsolicited data memory immediately. It remains to be seen what "a long time" is.

2.5.4. Example Handlers

Below is an example of a quick handler:

int my_LCB_HandlerQuick( LCB_msg *msg, void *usrData) {

int status = OK;

  if ( msg->error) {
    status = msg->error;
  }
  else {
    // our processing is quick, so just do it
    status = processData( usrData, msg->bulkData);
  }

  // free unsolicited data
  LCB_msgFree( msg);

  return status;
}

Below is an example of a slow handler, which makes a copy of the unsolicited data.

int my_LCB_HandlerSlow( LCB_msg *msg, void *usrData) {

int status = OK;

  if ( msg->error) {
    // free unsolicited data
    status = msg->error;
    LCB_msgFree( msg);
  }
  else {
    // our processing is slow, so make a copy using memcpy.
    // copy data to previously allocated memory, pointed to
    // by the "copy_mem" pointer.
    memcpy( copy_mem, (void *)msg, msg->length);

    // free the unsolicited data
    LCB_msgFree( msg);

    // process the copy
    status = processData( usrData, copy_mem);
  }

  return status;
}

3.2.1. Driver Initialization

Before beginning operation the polled mode LCBD requires several chunks of "scratch pad" memory, which must be provided by the boot application. The required memory sizes and characteristics are shown below:

The LCBD handle is used to manage the LCB.

A single command list is required to send LCB-to-LCB data on the event fabric.

A single result list is required because the LCB will generate a simple response to the bulk data transfer request.

The circular buffer is required to receive unsolicited data on the event fabric.

The function prototype for initializing the polled mode LCBD is shown below.

This call returns an initialized LCB handle. The cmdList parameter is a pointer to a memory 4KB memory chunk to use for the command list. The rstList parameter is a pointer to a memory 4KB memory chunk to use for the result list. The baseAddr parameter is the base memory address to use for the circular buffer.

At this point the LCB is completely configured. The reception of unsolicited data is enabled by calling LCB_PollStart() shown below.

3.2.2. Dispatching Results

This section leverages much of the machinery described previously for unsolicited data handling, see Section 2.5. The major difference is that the interrupt handler and callback routines are replaced with a simple polling loop.

The LAT boot application will poll the LCBD looking for new unsolicited data by calling the LCB_PollQuery() function.

When new LCB-to-LCB data is available this function returns a non-NULL LCB_msg pointer. When no data is available LCB_PollQuery() returns NULL.

Decoding the LCB_msg pointer proceeds as previously described in sections Section 2.5.2, Section 2.5.3 and Section 2.5.4. The big difference is that the decoding occurs within the polling loop, not in a user callback routine. See Section 3.3 for an example of the polled mode interface.

3.2.3. Sending Bulk Data

Sending bulk data on the event fabric is very straight forward. The maximum length of a bulk data item is 4080 bytes. Most of the discussion on sending bulk data using the ISR driver applies to the polled mode driver as well (see Section 2.4.1.2.4). The major difference is that the polled mode LCBD can only queue a single bulk transfer at a time.

The LCB_PollSend() function is used to send bulk data.

3.3.1. Initialization

An example of initializing the polled mode LCBD follows:

// global LCB handle
static LCB                  *lcb; // LCB handle

int initLCB() {

  int                status  =  LCB_OK;       
  void            *pCmdList; // command list
  void            *pRstList; // result  list
  void           *pBaseAddr; // base address of circular buffer

  // allocate memory for the command list -- suitably aligned
  pCmdList = mallocCommandList( 4096);

  // allocate memory for the result  list -- suitably aligned
  pRstList = mallocResultList( 4096);

  // allocate memory for the LCB handle
  lcb = malloc( LCB_SizeOf());

  // assign base address for circular buffer
  pBaseAddr = LCB_CIRC_BASE_ADDR;

  // initialize the LCB handle
  status = LCB_PollInit( lcb, pCmdList, pRstList, pBaseAddr);
  if ( status != LCB_OK) {
    free(lcb);
    free(pRstList);
    free(pCmdList);
    return status;
  }

  // the LCB is now initialized, but not enabled

  return status;

}
        

3.3.2. Polled Event Loop

This section describe the polled event loop. First the reception of unsolicited data is enabled and then the driver enters an infinite loop waiting for LCB messages. As messages are found they are processed.

int eventLoop( unsigned int usrData) {

  int                 status = LCB_OK;       
  LCB_msg             msg;

  // enable the LCB for receiving event data
  status = LCB_PollStart( lcb);

  if ( status != LCB_OK) {
    return status;
  }

  // enter infinite loop
  while ( 1) {
    
    if ( msg = LCB_PollQuery( lcb)) {
      // message found, process it.
  
      if ( msg->error) {
        // record/report error
        status = msg->error;
      }
      else {
        // proces the message.  The user could also make a copy of 
        // the bulk data at this time.
        status = processData( usrData, msg->bulkData);
      }
  
      // free unsolicited data
      LCB_msgFree( msg);

    }  

  // sleep, or poll other devices, etc...

  }

  return status;

}
        

3.3.3. Sending Bulk Data

This section describes sending bulk data on the event fabric.

int sendData( LCB *lcb) {

  int           status = LCB_OK;
  unsigned int  nBytes = 4000;   // number of bytes to send
  unsigned int  *data;
  short         destAddr = 0x41  // destination LATp node address
  short         proto = 3;       // LATp prototcol to use

  data = (unsigned int *)malloc( nBytes);

  // fill *data with interesting data to send

  status = LCB_PollSend( lcb, destAddr, proto, nBytes, data);

  free( data);

  return status;
}