U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Information Technology Laboratory / Information Access Division

Multimodal Information Group

OpenKWS15 Dry Run Instructions

Share

The following is the OpenKWS Evaluation Infrastructure setup and testing instructions using the Indus evaluation server.

  1. Download the IARPA-Babel Vietnamese Build and Eval packs with using the username and password provided by NIST.
  2. Download the latest IndusDB from OpenKWS Data Page.
  3. Download and install SCTK Version 2.4.10 or later (from http://www.nist.gov/itl/iad/mig/tools.cfm) according to the INSTALL document in the distribution.
  4. Download JobRunner 1.3.0 or later (from http://www.nist.gov/itl/iad/mig/tools.cfm) 
  5. Download and check the F4DE 3.2.0 or later (from http://www.nist.gov/itl/iad/mig/tools.cfm) according to the instructions under "INSTALLATION" section in the"README" file of the distribution.
  • Note, F4DE requires many CPAN libraries and other utilities.  It is vital that you successfully run either the full 'make check' or minimally the 'make mincheck' and 'make KWSEvalcheck' before proceeding to Step 6.
  • Send NIST Information to set up your Indus Scoring Server account which uses SSH public/private keys for authentication. (Instructions summarized from the "KWSEval/BABEL/Participants/README" in the F4DE release).
    1. If not familiar with SSH, please see: http://www.wikihow.com/Use-SSH
    2. Decide on the user name of your team; it must follow the pattern "kws-<team>[_<site>]" (<team> or <site> can not contain - or _)
    3. Generate an SSH Protocol 2 Key: Either ensure you have a valid SSH Protocol 2 public/private keys or generate a pair using either '% ssh-keygen -t dsa' or '% ssh-keygen -t rsa' with a non empty passphrase of 12 or more characters.
    4. Email openkws-poc [at] nist.gov (openkws-poc at nist.gov) with the following:
      1. your user, site and team names
      2. the list of users expected to use the service with contact information (email + phone)
      3. Attach your public key (not inline text) to the email
      4. Send a list of the IP subnets that will need access to the ssh Indus server
  • After completing step 6 and receiving your Indus Account, follow the Indus Scorer Setup instructions  found in "KWSEval/BABEL/Participants/README" of the F4DE release.
  • Run the test submissions through the scoring server by:
    1. Change directory to the KWSEval/BABEL/Participants/Checks directory
    2. Execute: "./SubmissionHelper_checks.sh ../KWS14-dbDir/samples-checkfiles/Comps--107_____KWS14*"
  • Unpacked password-protected zip file (TestData.zip) to make sure you can read the delivered data. The password is "abc123". The file listing should be:

    • TestData
    TestData/dir1
    TestData/dir1/subdir1
    TestData/dir1/subdir1/Readme.txt
    TestData/dir2
    TestData/TopReadme.txt

    Data and informatics
    Created October 1, 2014, Updated June 2, 2021