
This directory contains version 0.6 of the NIST HUB-4 IE spoke
scoring pipeline package. The pipeline compares a reference file in
the "utf" format to a hypothesis file in the "utf" format.  It
consists of several pre-existing programs, together with some scripts
to glue the programs together: 



        REFERENCE(UTF)      HYPOTHESIS(UTF)
              |                    |
	      |			   |
              V			   V
        +------------+ 	     +-----------+
        |      	     | 	     | 	       	 |
        |  UTF_FILT  |       |  UTF_FILT |
        |            |       |           |
	|   NSGMLS   |	     |	NSGMLS   |
        |            |       |           |
	+------------+	     +-----------+
	      |                    |
	      | 		   |
              V                    V
        +------------+ 	     +-----------+
        |      	     | 	     | 	       	 |
        |  CSRFILT   | 	     | CSRFILT 	 |
        |            |	     |	         |
        |  RFILTER1  |       | RFILTER1  |
	|            |	     |	         |
	+------------+	     +-----------+
              | 		   |
              |    +----------+	   |
	      |    |          |	   |
              +--->|  SCLITE  |<---+
		   |          |
		   +----------+
		       	|
		    	|
			V
              +---------------------+
              | 	            |
              |      TALDWRAP	    |
	      | 	            |
              |   TALD3E_SM_EXPORT  |
	      | 	            |
	      +---------------------+
	       		|
			|
			V
              +---------------------+
       	      | 	            |
              |       MSWRAP	    |
	      | 	            |
              |     MUC_SCORER	    |
	      | 	            |
	      +---------------------+
	       		|
	       		|
	       		V

                     RESULTS


Three utf conversion utilities (IdF_utf_wrapper.pl, ctm2utf.pl, 
and utf2nerf.pl) are also included.

The programs utf_filt, csrfilt (tranfilt), sclite(sctk),
tald3e_sm_export(aldistsm), IdF_utf_wrapper.pl, utf2nerf and ctm2utf.pl 
were produced by NIST.

The original IE spoke pipeline was produced by MITRE.

The programs reep, taldwrap, mswrap, and MUC_scorer were produced by SAIC.

NOTE:  Except for tald3e_sm_export, the programs here are slightly
modified for the IE spoke.  Please do not attempt to use a version
that you already have installed on your system.


REQUIREMENTS (please also see the readme's for individual programs)

  1) perl, version 5

  2) nsgmls, available with the SP package.  Please see

       ftp://ftp.jclark.com/pub/sp

     (NOTE: nsgmls version 1.3 will not work with this scoring
      software.  V1.2.1, and probably V1.3.1 will work.)

  3) the SGMLS.pm Perl Module, available on the CPAN.  A copy 
     is also at:

      ftp://ftp.muc.saic.com/pub/tools/SGMLSpm-1.03ii.tar.gz

  4) UN*X shell scripting, with the usual commands, pwd, rm, etc.

  5) a C compiler


INSTALLING THE SOFTWARE

  1) assume that the directory containing this file is called $IEEVAL

  2) to build the software, cd to $IEEVAL, and do "make".

     NOTE:  The build script for sclite in the $IEEVAL/software/sctk
     subdirectory will ask whether you would like to use gnu diff and
     the CMU Cambridge SLM toolkit.  These are not required for the
     IEEVAL pipeline, so you may answer "no".

  3) The overall pipeline script is $IEEVAL/scripts/reep.  To run it,
     you really have to set the environment variable IEEVAL to be
     the directory containing this file.  If SGMLS.pm isn't already in
     a place where perl can find it, you also need to set the PERL5LIB
     variable to include SGMLS.pm's directory.



DOCUMENTATION

In addition to the documentation which comes with each of the programs
in $IEEVAL/software, there are man pages and README's in $IEEVAL/doc.



KNOWN PROBLEMS

1) The pipeline does not work when using some later versions of
   nsgmls.  With nsgmls from SP 1.2.1, it works.  With the nsgmls from
   SP 1.3, it does not.  Even when it works, nsgmls sometimes issues 
   errors about an unknown character set.

2) Non-ascii characters are converted to their hex representation in
   the scoring process.


EXAMPLE DIRECTORY

There is a set of test runs in the subdirectories $IEEVAL/example
subdirectory. You may cd to $IEEVAL/example and do a "make" to run
them all at once, or cd to each subdirectory and do "make" separately.

The tests are, by directory:

1) example:

   Scores a reference against an empty hypothesis (a hyp with no NE
   tags), a reference against itself, and a reference against a
   hypothesis that has exactly three correct NE tags.

2) example2:
   
   Scores a larger reference against itself.

3) h4e_97

   Scores a very large reference against itself.

4) nonascii

   Scores a reference with all ascii against a hyp containing a
   non-ascii char. 

5) overlap

   Scores a reference with a named entity that spans a region of 
   overlapping speech against a hypothesis with the same named
   entity.  The results show that overlap regions in the audio are not
   scored. 

6) premods

   Tests that differences only in the inclusion/exclusion of "the",
   "a", and "an" between the reference and hypothesis do not result in
   missed points.

7) h4e_97_base-rec

   Scores a reference with a hypothesis produced by running BBN's 
   "IdentiFinder" baseline tagger on the output of a baseline recognizer.

8) h4e_97_ref-t

   Scores a reference with a hypothesis produced by running BBN's 
   "IdentiFinder" baseline tagger on a transcript file produced from 
   the script "utf2nerf.pl".

The shell scripts in the first three subdirectories, which are named 
"run_example.sh", show one way of scoring a utf file against itself.  

NOTE: You must have IdentiFinder installed, with the environment variable
      IDFROOT set to its location, for the last two examples to work.)



QUESTIONS/COMMENTS:

  Please email

     douthat@gso.saic.com

  To reach all HUB4 IE spoke participants, mail to 

     hub4_ie_list@jaguar.ncsl.nist.gov

