NIST logo







Benchmarking Study of Algorithms & CAD Tools

for the Measurement of Change in Lung CT


Organized by the National Institute of Standards and Technology



The Biochange Challenge is a comparative study of algorithms and CAD tools for the measurement of change in lung lesions using CT data. The primary goal is to assess and encourage the reduction of bias and variance in lesion change measurement. We offer a level playing field for developers and users to apply their computational methods. The clinical CT lung data is selected from the publicly available Reference Image Database to Evaluate Therapy Response (RIDER) database, including Coffee Break series ( In addition we include NIST-generated CT phantom series. A secondary goal of the work is to develop appropriate methods and testing materials for assessing software performance on the biomedical change measurement problem. The study builds on the methodology of the smaller Biochange 2008 Pilot Study.

The persistent problem in this and similar studies is the limited amount of reliable sizing information in clinical imagery, to serve as the basis for ground truth. In the present study, the size or the change in size of the lung lesions is known for only a fraction of the cases. The amount of change is known for the RIDER coffee break series. For the phantom series the lesion size (and therefore change in size) is known. The RIDER  investigators have published approaches to estimating the uncertainty associated with lesion measurements, particularly the variance and bias.  In the Biochange Challenge we will examine various approaches and their applicability to the algorithms and CAD tools from participating sites.

The 96 lesion pairs make this the largest change study using publicly available CT imagery, of which we are aware. It should be noted that studies of various biometric tools (e.g., typically rely on data sets that are orders of magnitude larger; the same is true of the data sets used to qualify the RECIST criteria for measuring response to therapy. From the CT collections, we have selected a set of 96 lesion pairs that vary in estimated size, size change and in lung location. The maximum slice thickness is 2.5 mm, with the majority at or below 1.25 mm.  Generally, the paired series have the same reconstruction kernel.


We plan to compare the performance of algorithms and CAD tools with each other and with available ground truth using publicly available clinical lung CT series and phantom data.  In addition, we seek to evaluate the methodology & performance measures used to validate change analysis.  The present study, the Biochange'08 Pilot, and the VOLCANO'09 study have a similar structure.

The main item to be reported is the fractional change in size for each lesion. The submissions will be evaluated by NIST and the results of the comparison will be reported to the participants. NIST will publish a paper on the results. Each participating site will be offered co-authorship on the evaluation publication.

Benchmarking elements

  • NIST gathers pairs of CT scans from various sources: RIDER (Reference Image Database to Evaluate Response) clinical data from NCI, a physical phantom collection (with hundreds of "lesions") and a synthetic phantom collection (having 8 lesions) from NIST. The data includes pairs of scans with "No Change" conditions.
  • To assure low barriers to entry, the Experiment Description is available both in XML format and as a spreadsheet. The evaluation has 96 lesion pairs for comparison at 2 time points.
  • Participants apply their algorithms or CAD tools to measure change in the lung lesions. The main item to be reported is the change in size for each lesion. For segmentation algorithms that determine two lesion volumes, V1 and V2, the fractional change is (V2/V1 - 1).
  • NIST defines and applies the evaluation methodology and reports results to participants.

How to Participate in Biochange Challenge

  • Download and read the Biochange Challenge Protocol and email or fax a Statement of Interest to NIST.
  • Download the BiochangeChallengeSeries from the NIST FTP site and from the NBIA RIDER collection as described in the Protocol.
  • Run your change analysis algorithm or CAD tool in your lab on the validation data.
  • Report your change results in one of the required formats and send a Participation Agreement signed by the your team leader to NIST by January 18, 2011.
  • NIST will analyze the reported results, comparing them to the limited available ground truth as described in the Protocol. NIST will provide Participants with individual analysis of their results. We will publish the results of the evaluation, without publicly identifying individual scores by Participant.



Charles Fenimore
phone: 301-975-2428
fax: 301-975-5287