*****************
*** Spawing : ***
*****************
1/ Host Server spawns App server
Command-line arguments passed :
- application name
- port to listen to

2/ App server spawns Duplicators
Command-line argument passed :
- flow name
- size of one buffer as requested by client
- mmap file size (in number of buffers)
- input or output
- if input flow, the system to connect to
- port to connect or listen to


********************************
*** Management connections : ***
********************************

( == XMLRPC invocations)

Connection that can be established :

1/ Client to App server :
for basic client management :
- presentation,
- authentication,
for advanced client management :
- statistics
- network status
- network request ( client quit or run,...)
for flow management :
- flow inquiries
- open/close a flow
- flow-control
- flow statistics

2/ Client to Host server :
for network management :
- application discovery (console)
- application server run (console)

3/ App server to Host server :
for network management :
- application subscribing (if the app server is ran by hand)

4/ App server to App server (on another system)
for smartflow protocol management :
for network management :
- client start, by request.

5/ Host server to Host server :
for network management :
- application discovery
- application start ( ask to spawn app servers )


**********************
*** Signaling link ***
**********************

*** IS ON THE CRITICAL PATH ***

(Under Unix : unix socket, under windows, whatever local mecanism, 
 maybe upipes)

Between a duplicator and a client.
Is used to synchronize access to the mmapped flow file.

The mecanism must be connection-based (not point-to-point) :
1 duplicator serves N clients.

Very few data passes by this link between a client and a duplicator.
Threads must block on a read() on it.
The simple fact of beeing unblock mean something happened.
The read() return :
- >=1 : signal, the value can indicate the buffer number to use
        in the mmaped file if >=2 are used.
- 0   : connection closed on the other side
- -1  : error, interpret and react (can be only an INTR from a signal,
        but can also be worse)


*************
*** Flows ***
*************

*** IS ON THE CRITICAL PATH ***

On a local machine, uses the mmaped file.
between systems, uses TCP or UDP socket.
(No great gain in UDP)

They are made of fixed-size buffers. The data can be smaller, and the 
metadata are 
- the actual size.
- the timestamp.

Every other metadata must be dealt with specific flow objects who
will know how to handle them, in a stack of flow objects.
(When doing that, Flows objects must also create a new Buffer object type, 
that override the data access methods)

Into a client, flows are made of a ring-buffer of Buffers objects.
The size of this ring-buffer on a consumer client is defined as 
the HISTORY.

A the main thread can do several things with the HISTORY.
it can get one buffer from it, inquiry on how many buffer are ready,
(and thus how many are free). It can also increase its size.
(Reducing is possible but not immediate, so a given history size mean 
"this number at least is guarantied, they may be more". )


CLIENT
-------
- At least two threads : One main thread and one messaging thread.

---- Messaging thread :
- On startup, open connection to the app server.
- Provide an API for cummunication to the other threads, who masks
the threading complexity and synchro locks.
- Clients can register callbacks for certain types of messages. 
*** warning, they will be called in the message thread context !!! ***
- threads can also block until a particular message arrives.
(useful for flow control)

All message transit in XMLRPC. A message is represented by a C++ class,
restricted to a subset of type for attributes.
The content of the message are the value of the attributes.
User can define their own method within their messages.
The 'void invoke(void)' method must be defined to accept incoming messages
(no invoke() method mean message dropped w/ a warning). At invocation time,
the object will have the exact same state than the sender version.

---- Main thread :
All the user work will run in the main thread. 
The user is free to create his own threads, BUT he must take care of locks
and synchronisation himself, and do not try to block the library threads
(by destroying/canceling them from the manager, for exemple)
The callbacks are called into this thread as well, serialized : this way,
sharing a variable between threads do not requiere locking.
(Singletons are recomended)

The callbacks are dispatched via a select() mecanism, on an event loop,
and this is the recommended behaviour.

However, a "one buffer shot" will be provided, like in the previous version,
to help event loops interlacing (GUI event loops come in mind), with a 
non-blocking option.


---- Flow thread :
These one are simple.
Input : 
- wait on the signaling link. On signal
- pickup a buffer from the ring (which is a linked list of BUFFER objects)
- lock the buffer
- read the mmaped file and fill the buffer
- unlock the buffer
- send an acknoledge on the signaling link.

Output :
- wait on the signaling link. On signal,
- pickup a buffer from the ring (which is a linked list of BUFFER objects)
- lock the buffer
- write the mmaped file from the buffer
- unlock the buffer
- send an acknoledge on the signaling link.


Problematic cases here :

Slow consuming :  The main thread does not consume buffers fast enougth.
The flow thread cannot find an empty buffer to pickup.
In this case, it can either :
- block, waiting for one, (making the other consumer wait) or 
- overwriting the last written one, or 
- give up, and send back the acknowlege immediatly
- try to increase the size of the ringbuffer, thus creating more buffers.

Fast consuming :
The main thread will stay block most of the time, waiting for an filled 
buffer, while the flow thread will wait for the duplicator signal.

Fast providing :
The flow thread has to wait for the signal from the duplicator before
writing to the mmaped file again. So it will block on the signaling link.
During that time, the main thread will exhaust the ringbuffer, and then 
he can :
- block, waiting for one. (useful when the data aren't time critical)
- overwriting the last written one, or
- give up, and lost its packet, or 
- increase the size of the ring buffer.
This case is the worse one, it mean we'll have a structural problem into
the Smartflow 2.

Slow providing :
The "normal" size for the ring buffer for a provider is 2 buffers : 
The flow thread is supposed to be blocked most of its time waiting
for a buffer to become ready, this is the "IDLE" state for the whole
smartflow  2 system.


A "watermark" system is provided, that can be enabled by the user who 
develop a consumer.
A high and low watermark must be given, in number of buffers. (it must be within
the history size)

When the number of filled buffer in the history reach the high watermark,
the flow thread is blocked to prevent him to add more buffers. 
The main thread continue to take buffers out until the number of buffers go
under the low watermark, which unblock the flow thread.

Additionally, there is a flow control :
One can give the system an indication of "XOFF" for some 
buffer numbers in the history.
When the number of filled buffer in the history reach one of
these flow control limit, a flow control message is sent to the provider 
throught the messaging system (embedded in the message is the buffer 
number hit and the total number of buffer in the history) 
How does the provider respond is up to him. He can reduce or completly stop 
his buffers, or just don't care and continue his current rate.

When using this mecanism, users must ensure that at least one XON indication
is linked to a buffer number. The XON message is sent to the provider as well.
The provider has then to opportunity to use these flow control message as 
a [xon/xoff] facility. 


DUPLICATOR
-----------

- Provider : program who memcpy() TO the mmaped file.
             Max one/flow can exist (0 or 1)
- Consumer : program who memcpy() FROM the mmaped file.
             Can be multiple/flow (0 or N)
- Local : Use the mmap file to get data.
- Remote : Use a TCP socket to get data.


A duplicator can have :
1/ One Local Provider, N Local Consumer and N Remote Consumers
2/ One Remote Provider and N Local Consumers.

In the first case, the external socket is put in listen mode.
when a remote client connect, duplicator start to send the content of
the mmap file.

In the second case, the external socket is connected to the remote peer.
when connected, the duplicator read the data and fill the mmap file.

In both case, local consumers wait for the signaling link, copy their buffer
from the mmaped file and send back a ack.
the duplicator is in charge for collectind the acks before sending
further notifications to consumers.



Note :
The ACE library provide all the low-level mecanisms to mask plateform and OS-
specific details,but it also provides more general facilities, like the ring buffer,
an abstraction of the socket through templates and inheritance, that can be used
to reduce the Smartflow 2 complexity and increase reliability.
( at least if the ACE classes are working as documented... )

ACE is at http://www.cs.wustl.edu/~schmidt/ACE.html , documentation is at 
http://www.dre.vanderbilt.edu/Doxygen/Current/html/ace/index.html

