# README¶

##### Created by Omid Sadjadi, June 22, 2018 (updated June 20, 2019)¶

### How to use the SRE18/19 scoring script¶

# 1) Setting the Python environnement¶

The scoring script has been developed and tested in Python 3 (version 3.7+). Here, we provide an example of a requirements file for conda.

requirements.txt

In [ ]:

    name: sre19
    dependencies:
      - python=3.7
      - numpy=1.16.4

## a) Using conda manager¶

Conda is an open source package and environment management system. The Miniconda version comes with a python distribution and is very light.

- Download Miniconda (Python3) from: https://conda.io/miniconda.html
- Run the following commands (based on the previous requirements file example)

In [ ]:

    > conda env create -f requirements.txt
    > source activate sre19

## b) Alternatives¶

If you already have your own Python 3.7 environnement (using virtualenv for example), use the pip command to install the required packages (e.g., numpy). Please make sure to meet the above noted requirements.

# 2) Running the scoring script¶

The root directory contains the main script (i.e., sre_submission_scorer.py) and with the default configuration as per the SRE18/19 evaluation plan guidelines. It also contains a sample system output file (i.e., system_output/sre18_dev_system_output.tsv), which is formatted according to the guidelines given in the SRE18/19 evaluation plan. Currently, only the dev data has been released, therefore we only provide an example for scoring the dev set.

## a) Testing the scoring script on a hypothesized system output¶

A good test is to first run the scoring script with the provided sample system output as input. This will first validate the system output and then computes the equalized metrics specified in the SRE18/19 evaluation plan. The sample system output is located in system_output/ folder. It meets the SRE18/19 system output format requirements.

From the root directory, execute the following command:

In [ ]:

    > python3 sre_submission_scorer.py -o system_output/sre18_dev_system_output.tsv \
    -l /path/to/LDC2019E59/dev/docs/sre18_dev_trials.tsv -r /path/to/LDC2019E59/dev/docs/sre18_dev_trial_key.tsv

## b) Scoring your own system output¶

From the root directory, execute the following command:

In [ ]:

    > python3 sre_submission_scorer.py -o /path/to/system_output.tsv \
    -l /path/to/LDC2019E59/dev/docs/sre18_dev_trials.tsv -r /path/to/LDC2019E59/dev/docs/sre18_dev_trial_key.tsv

# 3) Legal notice¶

This software was developed at the National Institute of Standards and Technology (NIST/ITL/IAD/MIG) by employees of the Federal Government in the course of their official duties. Pursuant to Title 17 Section 105 of the United States Code this software is not subject to copyright protection and is in the public domain. Permission to freely use, copy, modify, and distribute this software and its documentation without fee is hereby granted, PROVIDED that this notice and disclaimer of warranty appears in all copies

This software is offered AS IS. NIST assumes no responsibility whatsoever for its use by other parties, and makes no guarantees and NO WARRANTIES, EXPRESSED OR IMPLIED, about its quality, reliability, fitness for any purpose, or any other characteristic.

We would appreciate acknowledgement if the software is used.