Benchmarking Study of Algorithms & CAD Tools
for the Measurement of Change in Lung CT
Organized by the National Institute of Standards and Technology
The Biochange Challenge is a comparative study of algorithms and CAD tools for the measurement of change in lung lesions using CT data. The primary goal is to assess and encourage the reduction of bias and variance in lesion change measurement. We offer a level playing field for developers and users to apply their computational methods. The clinical CT lung data is selected from the publicly available Reference Image Database to Evaluate Therapy Response (RIDER) database, including Coffee Break series (https://wiki.nci.nih.gov/display/CIP/RIDER). In addition we include NIST-generated CT phantom series. A secondary goal of the work is to develop appropriate methods and testing materials for assessing software performance on the biomedical change measurement problem. The study builds on the methodology of the smaller Biochange 2008 Pilot Study.
The persistent problem in this and similar studies is the limited amount of reliable sizing information in clinical imagery, to serve as the basis for ground truth. In the present study, the size or the change in size of the lung lesions is known for only a fraction of the cases. The amount of change is known for the RIDER coffee break series. For the phantom series the lesion size (and therefore change in size) is known. The RIDER investigators have published approaches to estimating the uncertainty associated with lesion measurements, particularly the variance and bias. In the Biochange Challenge we will examine various approaches and their applicability to the algorithms and CAD tools from participating sites.
The 96 lesion pairs make this the largest change study using publicly available CT imagery, of which we are aware. It should be noted that studies of various biometric tools (e.g.,http://face.nist.gov/) typically rely on data sets that are orders of magnitude larger; the same is true of the data sets used to qualify the RECIST criteria for measuring response to therapy. From the CT collections, we have selected a set of 96 lesion pairs that vary in estimated size, size change and in lung location. The maximum slice thickness is 2.5 mm, with the majority at or below 1.25 mm. Generally, the paired series have the same reconstruction kernel.
We plan to compare the performance of algorithms and CAD tools with each other and with available ground truth using publicly available clinical lung CT series and phantom data. In addition, we seek to evaluate the methodology & performance measures used to validate change analysis. The present study, the Biochange'08 Pilot, and the VOLCANO'09 study have a similar structure.
The main item to be reported is the fractional change in size for each lesion. The submissions will be evaluated by NIST and the results of the comparison will be reported to the participants. NIST will publish a paper on the results. Each participating site will be offered co-authorship on the evaluation publication.
How to Participate in Biochange Challenge