Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.


Search Publications by

Ian Soboroff (Fed)

Search Title, Abstract, Conference, Citation, Keyword or Author
Displaying 1 - 25 of 51

PSCR 2021: Social Media Incident Streams

October 1, 2021
Ian Soboroff
Monitoring social media for public safety is incredibly challenging. The TREC Social Media Incident Streams project collects social media during emergency events, annotates and labels it for public safety use, and provides a metrics-focused environment

PSCR 2021: Pecha Kucha Portfolio Overviews

September 28, 2021
John Beltz, Scott Ledgerwood, Roger Blalock, Joe Grasso, John S. Garofolo, Jesse Frey, Cara O'Malley, Fernando Cintron, Bill Fisher, Gema Howell, Yee-Yin Choong, Jack Lewis, Paul Merritt, Edmond J. Golden III, Ian Soboroff, Craig Connelly, Gary Howarth, Brianna Vendetti, Katelynn Kapalo, Margaret Pinson
PSCR Research Portfolio Leaders join their staff to provide an overview of the projects housed within their PSCR portfolio. Each portfolio overview is delivered in a traditional Pecha Kucha style presentation, dividing topics into 20 slides that when

TREC 2020 News Track Overview

May 21, 2021
Ian Soboroff, Shudong Huang, Donna Harman
The News track focuses on information retrieval in the service of help- ing people read the news. In 2018, in cooperation with the Washington Post1, we released a new collection of nearly 600,000 news articles, and crafted two tasks related to how news is

PSCR 2020_Social Media Incident Streams

October 29, 2020
Ian M. Soboroff
The ubiquity of mobile internet-enabled devices combined with wide-spread social media use during emergencies is posing new challenges for response personnel. In particular, service operators are now expected to monitor these online channels to extract

International Workshop on Deep Video Understanding

October 21, 2020
Keith Curtis, George Awad, Shahzad K. Rajput, Ian Soboroff
This is the introduction paper to the International Workshop on Deep Video Understanding. In recent years, a growing trend towards working on understanding videos (in particular movies) in a more deeper level started to motivate researchers working in

Overview of the NIST 2016 LoReHLT Evaluation

November 13, 2017
Audrey N. Tong, Lukasz L. Diduch, Jonathan G. Fiscus, Yasaman Haghpanah, Shudong Huang, David M. Joy, Kay Peterson, Ian M. Soboroff
Initiated in conjunction with DARPA's Low Resource Languages for Emergent Incidents (LORELEI) Program, the NIST LoReHLT (Low Re-source Human Language Technology) evaluation series seeks to incubate research on fundamental natural language processing tasks

Using Replicates in Information Retrieval Evaluation

August 2, 2017
Ellen M. Voorhees, Daniel V. Samarov, Ian M. Soboroff
This paper explores a method for more accurately estimating the main effect of the system in a typical test-collection-based evaluation of information retrieval systems, and thus increasing the sensitivity of system comparisons. Randomly partitioning the

Promoting Repeatability Through Open Runs

June 7, 2016
Ellen M. Voorhees, Shahzad K. Rajput, Ian M. Soboroff
TREC 2015 introduced the concept of ‘Open Runs’ in response to the increasing focus on repeatability of information retrieval experiments. An Open Run is a TREC submission backed by a software repository such that the software in the repository reproduces

Computing confidence intervals for common IR measures

December 9, 2014
Ian M. Soboroff
Confidence intervals quantify the uncertainty in an average and o↵er a robust alternative to hypothesis testing. We measure the performance of standard and bootstrapped con- fidence intervals on a number of common IR measures using several TREC and NTCIR

Overview of the TREC-2012 Microblog Track

June 2, 2014
Ian M. Soboroff, Iadh Ounis, Jimmy Lin, Craig Macdonald
The Microblog track examines search tasks and evaluation meth- odologies for information seeking behaviours in microblogging en- vironments such as Twitter. It was first introduced in 2011, address- ing a real-time adhoc search task, whereby the user

Building Better Search Engines by Measuring Search Quality

March 3, 2014
Ellen M. Voorhees, Paul D. Over, Ian Soboroff
Search engines help users locate particular information within large stores of content developed for human consumption. For example, users expect web search engines to direct searchers to web sites based on the content of the site rather than the site

Overview of the TREC 2011 Microblog Track

August 15, 2013
Ian M. Soboroff, Iadh Ounis, Craig Macdonald, Jimmy Lin
The Microblog track examines search tasks and evaluation methodologies for information seeking behaviors in microblogging environments such as Twitter. It was first introduced in 2011, addressing a real-time adhoc search task, whereby the user wishes to

Evaluating Real-Time Search over Tweets

December 10, 2012
Ian M. Soboroff, Dean P. McCullough, Jimmy Lin, Craig Macdonald, Iadh Ounis, Richard McCreadie
Twitter offers a phenomenal platform for the social sharing of information. We describe new resources that have been created in the context of the Text Retrieval Conference (TREC) to support the academic study of Twitter as a real-time information source

Overview of the TREC-2010 Blog Track

August 15, 2012
Ian M. Soboroff, Iadh Ounis, Craig Macdonald
The Blog track aims to investigate the information seeking behavior in the blogosphere. The track was initiated in 2006, and has used an incremental approach in tackling several search tasks by their level of difficulty. In TREC 2010, the track has

Information Retrieval on the Blogosphere

July 30, 2012
Ian M. Soboroff, Rodrygo Santos, Craig Macdonald, Richard McCreadie, Iadh Ounis
Blogs have recently emerged as a new open, rapidly evolving and reac- tive publishing medium on the Web. Rather than managed by a central entity, the content on the blogosphere—the collection of all blogs on the Web—is produced by millions of independent

Overview of the TREC 2011 Web Track

November 15, 2011
Ian M. Soboroff, Nick Craswell, Charles L. Clarke, Gordon Cormack
The TREC Web Track explores and evaluates Web retrieval technology over large collections of Web data. In its current incarnation, the Web Track has been active for two years. For TREC 2010, the track includes three tasks: 1) an adhoc retrieval task, 2) a

A Comparative Analysis of Cascade Measures for Novelty and Diversity

February 10, 2011
Ian M. Soboroff, Charles L. Clarke, Nick Craswell, Azin Ashkan
Traditional editorial effectiveness measures, such as nDCG, remain standard for Web search evaluation. Unfortunately, these traditional measures can inappropriately reward re- dundant information and can fail to reflect the broad range of user needs that

The Effect of Assessor Errors on IR System Evaluation

July 19, 2010
Ben Carterette, Ian Soboroff
Recent efforts in test collection building have focused on scaling back the number of necessary relevance judgments and then scaling up the number of search topics. Since the largest source of variation in a Cranfield-style experiment comes from the topics

Test Collection Diagnosis and Treatment

June 15, 2010
Ian M. Soboroff
Test collections are a mainstay of information retrieval research. Since the 1990s, large reusable test collections have been developed in the context of community evaluations such as TREC, NTCIR, CLEF, and INEX. Recently, advances in pooling practice as

Blog Track Research at TREC

June 1, 2010
Ian M. Soboroff, Craig Macdonald, Rodrygo Santos, Iadh Ounis
The TREC Blog track aims to explore information seeking behaviour in the blogosphere, by building reusable test collections for blog-related search tasks. Since, its advent in TREC 2006, the Blog track has led to much research in this growing field, and

Overview of the TREC-2009 Blog Track

May 3, 2010
Ian M. Soboroff, Craig Macdonald, Iadh Ounis
In TREC 2009, the Blog track has been markedly revamped with the use of a new and larger sample of the blogosphere, cal led Blogs08, which has a 13-month timespan covering a period ranging from 14th January 2008 to 10th February 2009, and the introdu c-

Overview of the TREC 2008 Blog Track

January 12, 2010
Ian M. Soboroff, Iadh Ounis, Craig Macdonald
The Blog track explores the information seeking behaviour in the blogosphere. The track was introduced in 2006 [1], with a main pilot search task, namely the opinion-finding task. In TREC 2007 [2], the track investigated two main tasks inspired by the

Overview of the TREC 2008 Enterprise Track

January 1, 2010
Ian M. Soboroff, Krisztian Balog, Nick Craswell, Arjen de Vries, Paul Thomas, Peter Bailey
The goal of the enterprise track is to conduct experiments with enterprise data that reflect the experiences of users in real organizations. This year, we continued with the CERC collection introduced in TREC 2007. Topics were developed in conjunction with

Is spam an issue for opinionated blog post search?

July 19, 2009
Ian M. Soboroff, Craig Macdonald, Iadh Ounis
In opinion-finding, the retrieval system is tasked with re- trieving not just relevant documents, but those that also express an opinion towards the query target entity. This task has been studied in the context of the blogosphere by groups participating