Writing NML Configuration Files.

Most of the options available in CMS are specified in a configuration file. There should be one line in the file for every buffer that is required and every process using CMS should have one line for every buffer it accesses. Currently there is no difference between a CMS configuration file and an NML configuration file. The files may contain only ASCII characters, and can be generated with any text editor.

Update 9-Apr-2004: See The NML Configuration Server for information on how this information can be determined at run-time and also for some extensions to this format.

Lines beginning with # or white space are comments.

Lines beginning with B specify information about a particular buffer and are called buffer lines.

Lines beginning with P specify information about how a particular process accesses a buffer and are called process lines.

Before writing the details of the buffer and process lines, you will need to review the major buffer and process types available.

Buffer and Process Types

Local processes can connect to buffers using GLOBMEM, SHMEM, FILEMEM or LOCMEM. Remote processes can connect using TCP, UDP or RPC.

GLOBMEM

GLOBMEM is intended for communication across a VME backplane between processors with minimal operating system intervention, which allows greater portability and interoperability than SHMEM, but users need to specify the addresses of memory to use and mutual exclusion is implemented with a set of locks in memory rather than with semaphores provided by the operating system. GLOBMEM may also be used with a Bit3 ISA or EISA to VME adapter.

SHMEM

SHMEM is intended for communications between tasks managed by the same operating system. The operating system allocates the memory to be shared so users do not need to specify one. Users have a choice of mutual exclusion techniques with include something similar to GLOBMEM's method, using an operating system semaphore or mutex, or disabling and enabling context switching or interrupts during the appropriate critical sections.

Mutual Exclusion for Shared Memory

One of the most significant ways of optimizing shared memory accesses is to choose a the fastest form of mutual exclusion that still allows the application to run reliably. Mutual Exclusion is used to prevent multiple tasks, threads, or processors from using the same resource at the same time. This is necessary when using shared memory to prevent a process from reading the shared memory while only part of the message has been written or vice versa. Here are the currently available options:(One of them can be placed on the buffer line after the shared memory key.)

"mutex=os_sem" This is the default. An operating system semaphore is created using the same key used for shared memory. This semaphore is taken before each read or write and released afterwards. If another process already has taken but not released the semaphore when the semaphore is taken the operating system puts the process to sleep until the semaphore is released. This is reliable, however the operating system calls to take and release the semaphore take from 100 to 500 microseconds depending on CPU and operating system to execute. This is a very long time compared to the time required to copy small messages (less than 10k) into the shared memory area.

"mutex=none" The library will make no attempt to provide mutual exclusion. This is a very dangerous option that is only recommended when some other form of mutual exclusion is implemented on top of NML. One example might be to use only a single semaphore for multiple buffers like this:

	. . .
	rcs_sem_wait( . . .);
	type1 = nml1->read(); // configured not to provide mutual exclusion internally 
	type2 = nml2->read(); // configured not to provide mutual exclusion internally 
	type3 = nml3->read(); // configured not to provide mutual exclusion internally 
	rcs_sem_post( . . .);
	. . .

"mutex=no_interrupts" If there is only a single processor running multiple tasks or threads, most operating systems will be prevented from switching tasks, or threads as long as interrupts are disabled. Since disabling and enabling interupts is often much faster than the semaphore calls, this can be an efficient means of mutual exclusion. Currently this option will only work under Lynx and VxWorks. Under Lynx it is necessary to run the process as root. You also need to know how long your system can tolerate having interrupts disabled.

"mutex=no_switching" Some operating systems allow you to disable task switching which can be used just as disabling interrupts above, except that it is a little gentler in that it only prevents other tasks from executing but allows hardware interrupts to execute. Currently this option will only work under Lynx and VxWorks. Under Lynx it is necessary to run the process as root.

"mutex=mao split" This is an experimental option. It provides mutual exclusion by splitting the buffer in half. At any given time one half is used for reading while the other half is used for writing. Only at the end of a write is a flag changed that switches which half is used for what. A small area at the beginning of the buffer is used for registering which processes are reading and which are writing at any given time. This allows many readers and one writer to access the buffer simultaneously without allowing any reader to get a partial message or require semaphore calls. However situations can occur where it is necessary to force one of the processes to wait, or return an error. When a process is forced to wait there is currently no provision for priority inheritance, which means that currently while most of the time this option is very fast compared to the others, it can occasionally take significantly longer or timeout.

FILEMEM

FILEMEM is useful primarily for debugging. Messages are read from and written to a file, or messages may be read from one file and written to another. Record locking is used to provide mutual exclusion. This is the least efficient method, however the output file can be used for post-mortem analysis and scripts can be written and used as the input file for unit testing. FILEMEM buffers must have neutral set to 1 and add "disp" to the bufferline to use the displayable method of neutral data encoding. The input file can be set by adding "in=<filename>" at the end of the bufferline or it defaults to waiting for input from stdin, the output file can be set with "out=<filename>" or it defauls to stdout.

LOCMEM

LOCMEM is useful when many modules are linked together in one thread of execution but you want to write them such that each module uses the NML API just as it would if it needed to communicate with the other modules running separately. There is no need for any mutual exclusion mechanism and memory is obtained with a simple malloc so the operating system will not exceed its limits for semaphores or shared memory segments.

RPC

RPC stands for "Remote Procedure Call". There are two incompatible types of RPC, although they are both intended for the same purpose. CMS uses ONC RPC which is available on more of the platforms we were interested in rather than the NCS RPC which was chosen by OSF and is distributed with Visual C++. RPC was the first remote connection method available to NML users. Originally RPC was required for its dispatch services and eXternal Data Representation interface but these features are now provided in CMS on top of TCP or UDP. NML users that select RPC will not need to use any of the RPC API (just as they would not use any of the sockets API if TCP or UDP were selected), however they can check to see that the servers are registered with rpcinfo.

Update 9-Apr-2004: RPC is now deprecated.

TCP

TCP stands for "Transmission Control Protocol". This is the recommended method for remote connection, for most applications. It is more reliable and can handle larger messages than UDP and is more widely available than RPC.

STCP

STCP is a simplified form of the software on top of TCP. It is useful for as an interface to languages other than C++/Java where creating a binary version of the NML message structure(s) is difficult and/or there is no version of the RCS library available. It allows that applications in that language to send and receive simple text strings with low-level socket functions, while C++ and Java applications still use the normal NML API. It must be used with the display neutral encoding method. This means "disp" or "xml" must be placed on the bufferline and "xdr" or "ascii" must be removed if present.

UDP

UDP stands for "User Datagram Protocol". UDP is sometimes faster than TCP since it eliminates some of the error checking but messages should not exceed 512 bytes.

Buffer Lines

Lines declaring buffers should have the following format:

 
B 	name	type	host	size	neut	RPC# 	buffer# max_procs [Type-spec data]

B is the literal character identifying the line as describing a buffer.

name is a string identifying the buffer.

type is a string which can currently be "SHMEM" for shared memory buffers used for communications between processes on the same CPU , "GLOBMEM" for global memory buffers on a VME or EISA backplane, "LOCMEM" for a communications between functions run within the same process (multi-threaded applications should never use LOCMEM), or "PHANTOM" for buffers that use phantom functions.

host is a the name of the host for the buffer. If remote processes need to access the buffer they will look for an NML server on this host.

size is size of the buffer in decimal.

neut should be either 0 or 1. 1 indicates that the data stored in the buffer should be formatted into a machine independent or neutral format. Remote processes can still access buffers with a O even when running on a processor using incompatible types because the NML server can encode and decode automatically. However, the buffer itself must be encoded(neut=1) if the buffer will be accessed locally by processors of incompatible types.

RPC# - A remote procedure call number. All buffers that will be accessed using the same server should have the same RPC#. RPC program numbers 0x20000000 to 0x30000000 are for users to define. You may use the rpcinfo command on UNIX workstations to determine which numbers are already being used. Currently, the RPC# here is still required as a place holder even if TCP or UDP were to be used instead. This allows for some backward compatibility with configuration files written before TCP or UDP were options, however this backward compatibility may be dropped in the future.

Update 9-Apr-2004: Use of RPC is now deprecated. A TCP port or UDP port should always be specified, so the RPC# will ignored. 0 is typically used as a placeholder for this parameter.

buffer# - A server for more than one buffer will use the buffer number to determine which buffer to access for each request. Give each buffer a unique buffer number.

max_procs- The maximum number of processes allowed to use this buffer.(If the total possible connections equals n then the connection number in the process lines may be 0 to n-1.)

The area following the max_procs number is used to store information relevant to particular types of buffers, or to set options that have defaults.

GLOBMEM buffers require a physical address. The physical address may be specified with 1 or more address equations of the form "???_addr=0xnnnnnnnn" where ??? may be a host name, process name or bus name.

SHMEM buffers expect an integer key in the type specific area. The key needs to be a unique shared memory key on your system. For many UNIX systems the command "ipcs" can tell you which shared memory keys are in use. All of the NML buffers using SHMEM on a particular host need to have different keys.

To configure the use of TCP or UDP instead of RPC for remote access to the buffer, add either "TCP=" or "UDP=" and a port number to the buffer line. Use a port number greater than 1024 since ports 1-1023 are reserved for processes with user ID 0. It is recommended that you select a port between 5000 and 32000 to minimize the chance of conflicting with another application.

To use an alternative neutral encoding scheme where the messages are converted to ascii or plain text strings, add the word "ascii" to the buffer line. Or add "disp" to use an neutral encoding scheme that produces text strings with look much better for display. (The "disp" is required for FILEMEM.)

Update 9-Apr-2004: Use of "ascii" encoding is deprecated. For compatibility with previous versions of NML "xdr" should no be explicitly placed on the bufferline. A warning may be printed if no method is explicitly mentioned. XDR stands for "External Data Representation" a standard used by RPC. Two new encoding options available are "packed" and "xml". See NML support for XML for information concerning xml. "packed" encoding is unique to NML, it is binary similiar to XDR but not compaible with it. It packs structures more tightly and avoids byte-endian conversions when not necessary by adding the original endian type to the message. "packed" is the new default.

Update 14-Jun-2004: "packedl64" can be used to store long and unsigned long types in 64 bits. This is only useful if used on a system that uses 64 bit longs and unsigned longs, such as Itanium systems. (See NML 64bit long notes.)

To set the input file for FILEMEM add "in=" and the filename to the buffer line. Without this FILEMEM will use stdin.

To set the output file for FILEMEM add "out=" and the filename to the buffer line. Without this FILEMEM will use stdout.

To force FILEMEM to limit the number of messages in the output file add "max_out=" and the maximum number of messages to store in the output file. When this limit is reached the file will be overwritten starting at the beginning.

To enable queuing of messages in the buffer, add the word "queue" to the buffer line.

To enable supplemental timing diagnostic information to be logged to the buffer, add the word "diag" to the buffer line.( See Supplementary NML Timing Diagnostics Tools.)

To create a second semaphore needed for blocking_reads, add "bsem=<key>" where key is a unique semaphore key on that host. The key should also be different thatn the shmem key(s).

To force backwards version compatibility, add "version=<version number>".

To force the server to send confirmation messages back to a remote writer add "confirm_write". This allows a remote writer to detect problems such as a full queue or if a write_if_read failed because the buffer was not read but it reduces performance.

To set the baudrate to use on an RS232 serial port add "baud_rate=n" where n is and integer baud rate. The baud rates available depend on your hardware and operating system. The default is 9600. This parameter has no effect on non-serial port communications.

To prevent GLOBMEM from using "taskLock()/taskUnlock()" under VxWorks add "NO_TASK_LOCK". Note: the taskLock() is needed when multiple tasks on the same processor access the same GLOBMEM buffer, to ensure deadlocks do not occur. This tag can also be placed on the process line to effect only that process rather than all processes accessing this buffer.

Process Lines

Process lines should have the following format:

P 	name	buffer	type	host 	ops 	server 	timeout master 	c_num

P is the literal character identifying the line as describing a process.

name is a string matching the process name passed to the CMS config function or the NML constructor.

buffer is the string matching the identifier used in a buffer line to which this process intends to connect.

type is a string which can currently be "LOCAL" , "REMOTE", or "PHANTOM".

host is the name of the processor that the process is expected to run on.

ops is a string indicating whether the process will read or write.

server should be either 0, 1, 2, or 3. A 1 indicates that this process is responsible for running a server for the buffer, a 2 indicates that the nml_start function should spawn a server for the buffer, and a 0 is for processes that will neither spawn nor run a server for the buffer.

Update 9-Apr-2004: 3 indicates that a server should be spawned immediately rather than waiting for nml_start to be called. Unlike the other options, the TCP port or UDP port can not be shared with any other buffers.

timeout is the time in seconds for the process to wait for the mutual exclusion semaphore or for an RPC call to return before an error is reported. A time can be specified in floating point and the timeout is measured as close as possible for the given platform and process type or "INF" to never timeout.

master is a flag which should be 0 or 1, to indicate whether this process should create and initialize this buffer. On some operating systems there must be one and only one master process, on some there can be multiple masters. The parameter is ignored for REMOTE processes. There must be atleast one master that is a local process started before non-master and remote processes are started.

Update 9-Apr-2004: On some operating systems even non-masters can be started first and they will create the buffer, making this parameter almost irrelavent. We may attempt to eliminate this behavior in the future. For maximum portability, setting only a single process to be the master and ensuring it is started first is still recommended.

c_num is the connection number for this process.c_num must be between 0 and n-1 where n is the number of total_connections specified on the buffer line.

The area of the process line after the connection number is used to choose some options that have defaults. These options can be added in any order.

To have a remote processes set the non-blocking flag on its socket, add "poll".

To have a remote process setup a subscription with a TCP server add, "sub=<subscription interval>". The subscription interval is given in seconds as a floating point number. Example: To setup subscription interval of 20 milliseconds add "sub=0.02".

To prevent GLOBMEM from using "taskLock()/taskUnlock()" under VxWorks add "NO_TASK_LOCK". Note: the taskLock() is needed when multiple tasks on the same processor access the same GLOBMEM buffer, to ensure deadlocks do not occur. This tag can also be placed on the buffer line to effect all processes accessing this buffer.

To have a process use an RS232 serial port for communications add "serialPortDevName=<name>" where name is a device name such as "/dev/ttyb" or "COM2:". In addition the process must either be a server or be REMOTE. This is available only on certain platforms. By default, the port will be set to 9600 baud, 8 data bits, 1 stop bit, no parity.

Example: NML Configuration File

This configuration file provides a possible configuration file for the rest of the examples in this document. On your system, at the very least the host names will have to be changed. Although ex_buf4 is never used by any of the examples, but it is included as an example of the use of GLOBMEM.(The tab-stops on your browser or text editor may not make the columns line up neatly. NML only needs "some" white space between the items so feel free to add/remove tabs and spaces as needed for better appearance.)

ex_cfg.nml
# ex_cfg.nml

# buffers:
# name		type	host	size	neut RPC# 	buffer#	 max_proc [type-spec]
#		GLOBMEM	host	size	neut RPC# 	buffer#	 max_proc phys_addr
#		SHMEM	host	size	neut RPC# 	buffer#	 max_proc key 
B ex_buf1	SHMEM	dopey	512	0    0	        1	 11 	101 TCP=5001 
B ex_buf2	SHMEM	dopey	512	0    0	        2	 4 	102 TCP=5001 
B ex_buf3	GLOBMEM	vx40	512	0    0	        3	 4 	vme_addr=0x4e00000 TCP=5002


# processes:
# name		buffer	type	host 	ops	server 	timeout	master 	c_num
P ex1_proc	ex_buf1	LOCAL	dopey 	W	0	INF 	1	0
P ex2_proc	ex_buf1	LOCAL	dopey 	W	0	INF 	1	1
P ex2_proc	ex_buf2	LOCAL	dopey 	W	0	INF 	1	1
P ex3_proc	ex_buf1	LOCAL	dopey 	R	0	INF 	1	2
P ex4_proc	ex_buf1	LOCAL	dopey 	W	0	INF 	0	3
P ex5_proc	ex_buf1	PHANTOM	dopey 	W	0	0.5 	1	4
P ex6_proc	ex_buf1	LOCAL	dopey 	W	0	0.5 	1	5
P ex8_proc	ex_buf1	REMOTE	rosie 	RW	0	10.0 	0	6
P ex8_svr	ex_buf1	LOCAL	dopey 	RW	1 	5.0 	1	8
P ex8_svr	ex_buf2	LOCAL	dopey 	RW	1 	5.0 	1	8
P ex9_svr	ex_buf1	LOCAL	dopey 	RW	1 	5.0 	1	9
P ex9_svr	ex_buf2	LOCAL	dopey 	RW	1 	5.0 	1	9
P ex10_svr	ex_buf1	LOCAL	dopey 	RW	1 	5.0 	1	10
P ex10_svr	ex_buf2	LOCAL	dopey 	RW	1 	5.0 	1	10

Last Modified: 9-Apr-2004

If you have questions or comments regarding this page or you would like to be notified of changes to the RCS library via email, please contact Will Shackleford at shackle@cme.nist.gov