OpenKWS 2014 Evaluation Infrastructure Setup and Testing Instructions
The following is the OpenKWS Evaluation Infrastructure setup and testing instructions
- Download the IARPA-Babel Vietnamese Build and Eval packs with using the username and password provided by NIST.
- Download the latest IndusDB 20140206 (Contact NIST for link).
- Download and install SCTK Version 2.4.8 or later (from http://www.nist.gov/itl/iad/mig/tools.cfm) according to the INSTALL document in the distribution.
- Download JobRunner 1.3.0 or later (from http://www.nist.gov/itl/iad/mig/tools.cfm)
- Download and check the F4DE 3.1.0 or later (from http://www.nist.gov/itl/iad/mig/tools.cfm) according to the instructions under "INSTALLATION" section in the"README" file of the distribution.
Send NIST Information to set up your Indus Scoring Server account which uses SSH public/private keys for authentication. (Instructions summarized from the "KWSEval/BABEL/Participants/README" in the F4DE release).
- Note, F4DE requires many CPAN libraries and other utilities. It is vital that a you successfully run either the full "make check" or minimally the 'make mincheck' and 'make KWSEvalcheck' before proceeding to Step 6.
After completing step 6 and receiving your Indus Account, follow the Indus Scorer Setup instructions found in "KWSEval/BABEL/Participants/README" of the F4DE release (Note that this has been updated in F4DE 3.1.0).
Run the test submissions through the scoring server by:
- If not familiar with SSH, please see: http://www.wikihow.com/Use-SSH
- Decide on the user name of your team; it must follow the pattern "kws-<team>[_<site>]" (<team> or <site> can not contain - or _)
- Generate an SSH Protocol 2 Key: Either ensure you have a valid SSH Protocol 2 public/private keys or generate a pair using either '% ssh-keygen -t dsa' or '% ssh-keygen -t rsa' with a non empty passphrase of 12 or more characters.
- email 'openkws-poc at nist.gov' with the following:
- your user, site and team names
- the list of users expected to use the service with contact information (email + phone)
- Attach your public key (not inline text) to the email
- Send a list of the IP subnets that will need access to the ssh Indus server
Unpacked password-protected zip file (TestData.zip) to make sure you can read the delivered data. The password is "abc123". The file listing should be:
- Change directory to the KWSEval/BABEL/Participants/Checks directory
- Execute: "./SubmissionHelper_checks.sh ../KWS14-dbDir/samples-checkfiles/Comps--107_____KWS14*"