Shared Memory Mechanism

Intro

The State Machine Rman Interface uses a shared memory mechanism and a supporting daemon (incorporated into Norman) to manage the different memory resources available on the State Machine Stream Processor. The shared memory segments created and accessed using this mechanism are used to specify aggregate parameter types (including processor state) as arguments to functions running on the State Machine processors. Internal to Norman, this same mechanism is used for allocating temporary buffers for stream input and output.

This shared memory mechanism has a programming interface similar to that of the UNIX shared memory library. Segments must first be created, usually by the application, which obtains a unique shared memory ID for the segment. This ID is then passed to other processes that need to access that memory structure, taking the place of the pointer used to identify "normal" memory data structures. Before using, a shared segment must be attached, which dereferences the ID into a pointer in the process's address space. A shared segment is aware of the number of threads currently attached (which includes references to it in queues currently waiting for execution.) When a process is finished with a memory segment, it is detached. A segment is either destroyed by its application, or garbage-collected when all processes have detached.

Memory Classes

Constant

The simplest storage class is SMEM_CONST, which is never changed after initialization. Multiple copies of memory segments with this class may be maintained in different stream processors.

Variable

The SMEM_VAR storage class is used for segments that store state (such as intermediate buffers used in transpose and remap operations). Only a single instance of memory segments of this class may exist.

Local Dynamic

The SMEM_LOCAL storage class represents a memory segment that only exists temporarily on the stream processor. No memory is allocated by the RmanCreate call - instead it serves to establish the size of the segment. The memory is allocated later, in the local memory of the appropriate stream processor, when the resource manager attaches to that segment in setting up a mini-queue transfer. It is used to allocate temporary buffers for stream input and output on the state machine stream processor.

The SMEM_VAR class bears a little more description. These are shared memory segments whose contents may modified. When a process attaches to these segments, it locks out all other processes. If another process is currently attached to this class of segment, an RmanAttach operation will return an error value indicating this condition.

The Norman resource manager will not schedule a pipeline for execution until all the shared memory segments in a function's parameter list have successfully been attached. The process of attaching to a shared memory segment may require that the current version of the segment be copied into the address space of the requesting processor, and invalidated. The copy is then marked as the current version and a pointer to it returned. The current version of a variable segment may remain on a stream processor until it is either attached by a process running on the P2 local processor or flushed due to a shortage of stream processor memory. The resource manager takes the current location of shared memory segments into account when assigning stream processors to pipelines.

Typical Usage

A typical usage for a Constant class segment is :

1. An application calls RmanCreate to allocate an SMEM_CONST class segment for an array of filter coefficients.
2. RmanAttach is called to obtain a pointer to the memory segment, and it is initialized.
3. RmanStateMach is used to build a TransferQueue that contains this shared segment's id in it's parameter list.
4. One or more RmanExecute calls are made with that TransferQueue. The resource manager, upon encountering a mini-queue ready for execution with a shared memory id, checks to see if that shared segment already exists on a stream processor, allocating memory on one and copying the shared memory segment into it if necessary.
5. If the application decides to modify the filter coefficients, it merely uses RmanDetach to detach the current set of coefficients and creates a new Constant shared segment, which is then attached, initialized and its ID inserted into the parameter structure. The old shared memory segment will be discarded upon completion of all operations that reference it.
6. The application, when done, may merely detach from a segment. It will be destroyed if no more processes are using it. It may also call RmanDestroy to immediately remove the segment.

A typical usage for a Variable class segment is :

1. An application calls RmanCreate to allocate an SMEM_VAR class segment for a histogram.
2. RmanAttach is called to obtain a pointer to the memory segment, and it is initialized to zero.
3. RmanDetach is called to indicate that the process is done modifying the segment.
4. RmanStateMach is used to build a TransferQueue that contains this shared segment's id in it's parameter list.
5. One or more RmanExecute calls are made with that TransferQueue. The resource manager, upon encountering a mini-queue ready for execution with a shared memory id, checks to see if that shared segment already exists on a stream processor, allocating memory on one and copying the shared memory segment into it if necessary.
6. If the application wants to read the histogram, or clear it to zero, it merely repeats steps 2, 3, and 5 above.
7. The application, when done, should call RmanDestroy to delete the segment.

C Declarations

Here are the C declarations for the shared memory interface. These defines and data structures, as well as prototypes for the functions, are automatically included into <cheops_rman.h>, and are actually defined in <sys/params.h>.

#define SMEM_ID_ERROR   0  /* not valid id - indicates error in create */
#define SMEM_PARAM( num )   ( 1 << (num)) /* shared mem parameter bit field */
typedef int smem_id;

typedef enum {
  SMEM_CONST,   /*  written once and not modified */
  SMEM_VAR,     /*  A changing variable, whose coherence must be maintained. */
  SMEM_LOCAL,   /*  localbuffer - not init'ed by normal means, and temporary */
} smem_class;

• unsigned long size - The size, in bytes, of the memory block to be created.
• smem_class class - The class of the memory block to be created.

This function creates a shared memory object, and returns an ID for it. This ID can be translated into a physical pointer to a local memory object which a processor can directly read or write using the RmanAttach() function.

In case the segment could not be attached (when another process has it locked, in the case of Variable class segments), the pointer returned is equal to zero (NULL).

This function takes a shared memory ID and deallocates any local resources it is using, but does not force the destruction of the actual shared memory object and the data stored in it.

Please send comments regarding this proposal to wad@media.mit.edu

This is a "fix it yourself" page, located at ${CHEOPS_BASE}/WWW/statemac/software/smem.html