U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Search Publications by: Ian Soboroff (Fed)

Search Title, Abstract, Conference, Citation, Keyword or Author
Displaying 26 - 50 of 131

Podcast Metadata and Content: Episode Relevance and Attractiveness in Ad Hoc Search

July 11, 2021
Author(s)
Ian Soboroff, Ben Carterette, Rosie Jones, Gareth Jones, Maria Eskevich, Sravana Reddy, Ann Clifton, Yongze Yu, Jussi Karlgren
Rapidly growing online podcast archives contain diverse content on a wide range of topics. These archives form an important resource for entertainment and professional use, but their value can only be realized if users can rapidly and reliably locate

TREC Deep Learning Track: Reusable Test Collections in the Large Data Regime

July 11, 2021
Author(s)
Ellen M. Voorhees, Ian Soboroff, Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos
The TREC Deep Learning (DL) Track studies ad hoc search in the large data regime, meaning that a large set of human-labeled training data is available. Results so far indicate that the best models with large data are likely deep neural networks. This paper

Evaluating Multimedia and Language Tasks

May 24, 2021
Author(s)
Ian Soboroff, George Awad, Asad Butt, Keith Curtis
Evaluating information access tasks, including textual and multimedia search, question answering, and understanding has been the core mission of NIST's Retrieval Group since 1989. The TRECVID Evaluations of Multimedia Access began in 2001 with a goal of

TREC 2020 News Track Overview

May 21, 2021
Author(s)
Ian Soboroff, Shudong Huang, Donna Harman
The News track focuses on information retrieval in the service of help- ing people read the news. In 2018, in cooperation with the Washington Post1, we released a new collection of nearly 600,000 news articles, and crafted two tasks related to how news is

TREC-COVID: Constructing a Pandemic Information Retrieval Test Collection

February 19, 2021
Author(s)
Ellen M. Voorhees, Ian Soboroff, Tasmeer Alam, William Hersh, Kirk Roberts, Dina Demner-Fushman, Kyle Lo, Lucy L. Wang, Steven Bedrick
TREC-COVID is a community evaluation designed to build a test collection that captures the information needs of biomedical researchers using the scientific literature during a pandemic. One of the key characteristics of pandemic search is the accelerated

PSCR 2020_Social Media Incident Streams

October 29, 2020
Author(s)
Ian M. Soboroff
The ubiquity of mobile internet-enabled devices combined with wide-spread social media use during emergencies is posing new challenges for response personnel. In particular, service operators are now expected to monitor these online channels to extract

International Workshop on Deep Video Understanding

October 21, 2020
Author(s)
Keith Curtis, George Awad, Shahzad K. Rajput, Ian Soboroff
This is the introduction paper to the International Workshop on Deep Video Understanding. In recent years, a growing trend towards working on understanding videos (in particular movies) in a more deeper level started to motivate researchers working in

TREC-COVID: Rationale and Structure of an Information Retrieval Shared Task for COVID-19

July 8, 2020
Author(s)
Ellen M. Voorhees, Ian Soboroff, Tasmeer Alam, Kirk Roberts, William Hersh, Dina Demner-Fushman, Steven Bedrick, Kyle Lo, Lucy L. Wang
TREC-COVID is an information retrieval (IR) shared task initiated to support clinicians and clinical research during the COVID-19 pandemic. IR for pandemics breaks many normal assumptions, which can be seen by examining nine important basic IR research

Overview of the NIST 2016 LoReHLT Evaluation

November 13, 2017
Author(s)
Audrey N. Tong, Lukasz L. Diduch, Jonathan G. Fiscus, Yasaman Haghpanah, Shudong Huang, David M. Joy, Kay Peterson, Ian M. Soboroff
Initiated in conjunction with DARPA's Low Resource Languages for Emergent Incidents (LORELEI) Program, the NIST LoReHLT (Low Re-source Human Language Technology) evaluation series seeks to incubate research on fundamental natural language processing tasks

Using Replicates in Information Retrieval Evaluation

August 2, 2017
Author(s)
Ellen M. Voorhees, Daniel V. Samarov, Ian M. Soboroff
This paper explores a method for more accurately estimating the main effect of the system in a typical test-collection-based evaluation of information retrieval systems, and thus increasing the sensitivity of system comparisons. Randomly partitioning the

Promoting Repeatability Through Open Runs

June 7, 2016
Author(s)
Ellen M. Voorhees, Shahzad K. Rajput, Ian M. Soboroff
TREC 2015 introduced the concept of ‘Open Runs’ in response to the increasing focus on repeatability of information retrieval experiments. An Open Run is a TREC submission backed by a software repository such that the software in the repository reproduces

Computing confidence intervals for common IR measures

December 9, 2014
Author(s)
Ian M. Soboroff
Confidence intervals quantify the uncertainty in an average and o↵er a robust alternative to hypothesis testing. We measure the performance of standard and bootstrapped con- fidence intervals on a number of common IR measures using several TREC and NTCIR

Overview of the TREC-2012 Microblog Track

June 2, 2014
Author(s)
Ian M. Soboroff, Iadh Ounis, Jimmy Lin, Craig Macdonald
The Microblog track examines search tasks and evaluation meth- odologies for information seeking behaviours in microblogging en- vironments such as Twitter. It was first introduced in 2011, address- ing a real-time adhoc search task, whereby the user