1. CTS Echo Cancellation (02/28/03)

The CTS evaluation test material was distributed WITHOUT echo cancellation.  
This is a change to our previous practice and is intended to permit you to 
use the algorithm of your choice.  If you wish to use the algorithm that NIST 
has traditionally used in preparing training and test material in the past, 
please use the echo cancellation software available from the Mississippi State 
archive:

http://www.isip.msstate.edu/projects/speech/software/legacy/fir_echo_canceller/index.html

To keep the playing field level, you need not count echo cancellation 
in your realtime calculation.  If you run it during recognition processing, 
the  "official" realtime calculation you report should  be (your total processing 
time as per the evaluation plan - echo cancellation processing time) divided by 
the recording duration.


2. RT-03S Processing Speed Computation (03/13/03)

Total Processing Time (TPT):

For this and future RT evaluations, the time to be reported is the Total Processing Time (TPT) 
it takes to process all channels of the recorded speech (including ALL I/O) on a single CPU.   
TPT represents the time a system would take to process the recorded audio input and produce 
lexical token output as measured by a stopwatch.  So that research systems that aren't completely 
pipelined aren't penalized, the "stopwatch" may be stopped between (batch) processes.  Note that 
TPT should exclude time to implement CTS echo cancellation.   This is so that sites using the 
Mississippi State Echo Cancellation Software, which was not optimized  for speed or integration, 
are not penalized.  TPT may also exclude time to "warm up" the system prior to loading the test 
recordings (e.g., loading models into memory.)

Source Signal  Duration (SSD):

In order to calculate the realtime factor, the duration of the source signal recording
must be determined.  The source signal duration (SSD) is the actual recording
time for the audio used in the experiment as specified in the experiment's UEM 
files.  This time is channel-indepenendent and should be calculated across all 
channels for multi-channel recordings.  

Speed Factor (SF) Computation:

The speed factor (SF) (also known as "X" and "times-realtime") is calculated as follows:

    SF = TPT/SSD

For example, a 1-hour news broadcast processed in 10 hours would have a SF of 10 (regardless
of whether the broadcast is stereo or monaural).  And a 5-minute telephone conversation processed
in 50 minutes would also have an SF of 10 (regardless of whether the signal is a 4-wire/2-channel
signal or a 2-wire/1-channel signal).

Reporting Your Processing Speed Information:

Although we encourage you to break out your processing time components into as much detail 
as you like, you should minimally report the above information in the system description for
each of your submitted experiments in the form:
TPT = <FLOAT>
SSD = <FLOAT>
SF = <FLOAT>