Link to the Information Technology Laboratory Website Link to the Information Access Division Website Link to the NIST Website NIST, IAD Banner

The Buffer class

Once you have successfully created a flow, you can get a Buffer from your flow object using the method Flow::getBuffer(). You should never create or destroy a Buffer; instead, the Flow object takes care of that for you. The same method is used to get a Buffer for input or output flow:

  • for an output flow, this method returns an empty Buffer ready to be filled with data.

  • for an input flow, this method returns a Buffer filled with data. If there is no provider, the method is blocking. It only returns when a Buffer is available. This behavior can be annoying in some applications, which try to get a Buffer but don't want to be blocked if no data is available.

Be aware that the Buffer does not handle endianness concerning your data. It can be done by some specific Flows for specific kind of data.

You can test whether a Buffer is available by calling the Flow::isBufferAvailable() method. Depending on the result of this method, you can decide whether to ask for a Buffer or not. See the Avoiding blocking if no data is available section fore more details.

In order to use the Buffer object, you must include the following header in your code:

#include "Buffer.h"

Once you get you your Buffer, there are several ways to fill it with data or reads its content. The way you read your Buffer obtained from an input flow depends on how the Buffer has originally been filled.

The memcpy approach

This approach is the simplest one. It is really convenient to use if you want to transfer one chunk of data. Consumer and producer client nodes using this approach are provided to illustrate this concept.

They are located in the src/clients/example_provider_memcpy and src/clients/example_consumer_memcpy folders.

How to fill the Buffer

All you need to do is to copy your data in the writing area of the Buffer. You can access this memory area using the method Buffer::data(). It returns a pointer on the beginning of the writing area of the Buffer. Using the memcpy() method is essentially all you need to fill out the Buffer:

ACE_OS::memcpy(myBuffer->data(), &myData, sizeOfMyData);

Here &myData is the address of your data and sizeOfMyData its size. We recommend that you use the ACE::memcpy() method for portability issues. After copying your data in the Buffer, you need to specify how much data has been copied because the Buffer object has no way to determine how much data has been has been written. You can do this as follows:

myBuffer->setSize(sizeOfYourData);

The Buffer is now ready to be sent to any client node that subscribed to the flow. You send the Buffer by releasing it. Releasing a Buffer tells the system that you have finished working on it. The systems therefore can send it to the consumers that subscribed to the flow. If there is no consumer for the flow, the Buffer is just dropped.

flow->releaseBuffer(myBuffer);

This call frees the memory allocated to the Buffer so you don’t have to handle it yourself.

How to read the Buffer

On the consumer side, here is how you read a Buffer filled using the memcpy approach.

Before copying the data from the NDFS Buffer object to your local buffer, you need to allocate some memory. A specific method is provided to let you know how much space you need.

myBuffer = flow->getBuffer(); 
bufferSize = myBuffer->getSize();

At this point, you know the size you need to copy your data from the buffer to your object.

ACE_OS::memcpy(&myData, myBuffer->data() , bufferSize);

In the example above, &myData is the address of the object where you want to copy your data, myBuffer->data() is the address of the writing area of the Buffer and bufferSize the size of the data contained in the buffer.

Once you have copied your data, the buffer is no longer needed so you can release it.

flow->releaseBuffer(myBuffer);

The memcpy approach is the lowest level approach to access the data blocks. In that case, the data you receive are exactly the same you sent in the byte order. If your sender and receiver nodes are running on different kinds of operating systems or architectures, you will need to handle 32bits vs. 64bits or endianness issues yourself.

The pushData/popData approach

This approach uses methods from the NDFS II API to fill the Buffer with data. It is inspired by the push/pop methods of the C++ STL containers. The NDFS-II Buffer can be seen as a generic array, which can be filled with data blocks using our API methods. A Buffer is then an array of data chunks, which can be pushed or popped. The buffer is not tied to a specific data type, meaning that you can successively add a combination of different kinds of data and metadata. Client nodes are provided to highlight these approach and are located in the src/clients/example_provider_pushdata and src/clients/example_consumer_popdata folders.

How to fill the Buffer

As in the memcpy approach, you first need to get a buffer:

myBuffer = flow->getBuffer();

Before filling it, you have to tell the Buffer object you will use the pushData approach to fill the Buffer. In order to do that, you must call:

myBuffer->initBuf();

Then you can fill it as follows:

myBuffer->pushData(&dataToSend, myDataSize);

You can push as much data as you want until the Buffer is full. The pushData() method returns a boolean value letting you know if the data has been properly copied.

When you have filled the buffer, you just need to release it in order to send it.

flow->releaseBuffer(myBuffer); 

How to read the Buffer

Symmetrically, as when you fill up a buffer, you need first to get a buffer from a flow.

myBuffer = flow->getBuffer();

Then, after allocating some space to store your data, you need to pop the data from the buffer.

myBuffer->popData(&myData, myDataSize);

After popping the data, the NDFS buffer is no longer required, so you can just release it.

flow->releaseBuffer(myBuffer); 

Data Vs. Metadata

Buffers can be filled with a combination of data and metadata in any order. The only constraint is the size of the Buffer.

Each time you send a Buffer, or you push a data chunk in a buffer, a timestamp can automatically be added. The timestamp used is the one provided by the operating system. You can however add a timestamp coming from a device or one you generate yourself. Here are the different methods available to add data or metadata in a Buffer.

bool pushData(void *src, buffer_size_t size);

bool popData(void *src,buffer_size_t size);

The first of the two methods above is used to push data chunks in a Buffer. src is the address of your data and size its size. These methods work together.

bool pushDataTS(void *src, buffer_size_t size, ACE_Time_Value *ts = 0);

bool popDataTS(void *src,buffer_size_t size, ACE_Time_Value *ts );

These two methods are the same as the pushData() and popData() methods but they give the ability to provide and retrieve a timestamp associated with the data block. The third parameter ts of the pushDataTS() is optional. If you decide to provide your own timestamp, which has to be an ACE_Time_Value timestamp, it will be transmitted along with the data chunk. If you don’t, the timestamp associated with the data chunk is generated by the operating system. When using pushDataTS() on the provider side, you should use popDataTS() on the consumer side.

bool pushMetadata(void *src, buffer_size_t size, data_type_t type = 0);

bool popMetadata(void *src,buffer_size_t size );

These methods are used to push and pop metadata in the Buffer. src is the address of your metadata. It can be of any kind with any size as long as it could fit in the Buffer. Conceptually, data or metadata are binary data. So metadata can be seen as data associated with a flag specifying that it is actually metadata. The optional type parameter specifies the metadata type. If you don’t provide the third parameter, the default metadata type will be used. You may need to handle several metadata types so the data flow system allows you to create and provide your own type of metadata in order to make a difference between your metadata kinds. More information and examples about custom metadata can be found in the Creating your own metadata type section.

All these methods return a boolean indicating if the data or metadata has been properly copied in the Buffer. Using these methods allows to add in a buffer for example a video frame followed by two metadata providing the name of the codec used or some relevant information associated to it. Two examples including a producer and a consumer are provided and located in the src/clients/example_provider_push_data and src/clients/example_consumer_popdata folders.

Combining an iterator with data and metadata

The NDFS-II API provides a Buffer iterator to ease the data retrieval from the Buffer object, which has been filled with multiple pushData() calls. Here is sample code to show how to instantiate an iterator and associate it with a Buffer:

BufferIterator *iterator = new BufferIterator();
myBuffer = flow->getBuffer();

//We got the Buffer from the flow and we now are associating it to the iterator
iterator->setBuffer(myBuffer);
iterator->init();

while ( !iterator->end() ) {
      switch(iterator->getDataType()) {
      case DATA:
         //We received Data
         dataSize = iterator->getDataSize();
         //memory allocation of myData using the proper type and the size
         //‘dataSize’ needs to be done before getting the data from the Buffer
         .....
         iterator->getData(myData);
         break;
      case METADATA:
        //We received Metadata
        metaDataSize = iterator->getDataSize();
        //memory allocation of myMetaData using the proper type and the size
        //‘metaDataSize’ needs to be done before getting the metadata from the
        //Buffer
        .....
        iterator->getData(myMetaData);
        break;
      case DATAWITHTIMESTAMP:
        //We received Data with a timestamp
        dataSize = iterator->getDataSize();
        //memory allocation of myData using the proper type and the size
        //‘dataSize’ needs to be done before getting the data from the Buffer
        .....
        iterator->getData(myData);
        ACE_TIME_VALUE ts = iterator->getTimestamp();
        break;
      default:
        break;
}

In this code, we get a buffer from the flow, create an iterator and associate it with the buffer. The iterator can be reused later with a new buffer by invoking the setBuffer() and init() method again. Then we iterate on our buffer using the iterator as long as we have data in the buffer. The getDataType() method returns the data type of the current data chunk we are presently iterating on. Using that piece of information, we can get the data using the method getData() of the iterator. If a timestamp is associated with the data block, we can get it using the getTimestamp() method. If no timestamp has been associated with the data chunk and the method is invoked anyway, the timestamp returned has its fields initialized with the 0 value. Examples to highlight this approach are provided and can be found in the src/clients/example_provider_multipush and src/clients/example_consumer_iterator folders.

Creating your own metadata type

The METADATA type is a generic type provided by the system. Many multimodal applications may require the use of several metadata types. So users can declare their own types of metadata. The following is an example of how to declare this custom type:

#define NEWMETADATA 10
#define OTHERMETADATA 11

A metadata type only needs representing a metadata type ID. This declaration must be in both consumer and provider client nodes or could be declared in a custom flow and is done using an integer. The integer associated with the type must be superior to 10. Values under 10 are reserved for internal use.

On the provider part, putting a custom metadata in the buffer will look like:

mybuffer->pushMetadata(&theMetaData, metadataSize, NEWMETADATA);

On the consumer part, the following code needs to be included in the while loop shown in the Combining an iterator with data and metadata section:

case NEWMETADATA: 
     //We received NEWMETADATA 
     metaDataSize = iterator->getDataSize(); 
     //the memory allocation of myMetaData using the proper type and the 
     //size ‘metaDataSize’ needs to be done before getting the metadata from
     //the Buffer 
     .... 
     iterator->getData(myMetaData);
     break;

Using custom metadata can be very useful especially when a consumer does not exactly know what it will get from a provider. One can for example push different kinds of data in a buffer. If metadata information is pushed in the Buffer before pushing any chunk of data, it is then possible to know what kind of data is coming and act accordingly. See the example_consumer_custom_metadata and example_provider_custom_metadata located in the src/clients folder for a full example.

Created on 2008-06-18 by Antoine Fillinger - Last updated on 2008-11-23 by Antoine Fillinger