Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Reproducibility of Imaging Analyses Applied to Nucleus Image Quantification


Our motivation for addressing the variability problem in image-based measurements comes from increasing reports of irreproducibility in the Artificial Intelligence/Machine learning (AI/ML) field. Most recently, two new analyses [1, 2] put the spotlight on machine learning in health research, where lack of reproducibility and poor quality could risk harm to patients and/or lower the quality of care a patient receives. There is a need to quantify and minimize measurement uncertainties from various computational sources while leveraging all cutting-edge AI/ML approaches to image-based drug discoveries.

[1] M. B. A. McDermott, S. Wang, N. Marinsek, R. Ranganath, L. Foschini, and M. Ghassemi, “Reproducibility in machine learning for health research: Still a ways to go,” Sci. Transl. Med., vol. 13, no. 586, Mar. 2021, DOI: 10.1126/scitranslmed.abb1655.

[2] M. Roberts et al., “Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans,” Nat. Mach. Intell., vol. 3, no. 3, pp. 199--217, Mar. 2021, DOI: 10.1038/s42256-021-00307-0.


The goal of our effort is to improve reproducibility of image-based measurements and quantify the measurement uncertainty while leveraging all cutting-edge AI/ML approaches to image-based drug discoveries. In our work, we selected measurements derived from fluorescently-labeled nucleus images over samples with a variety of drug treatments and imaging variations. Such image-based measurements require

  1. nucleus segmentation from background,
  2. labeling each nucleus with a unique label, and
  3. computing intensity and shape characteristics per unique label of a nucleus.

The challenges of delivering reproducible nucleus measurements lie in a lack of

  • interoperability of cutting-edge approaches and their implementations,
  • automatic capturing of computational provenance about input and intermediate datasets, as well as about parameter settings, for sharing across teams and institutions,
  • understanding of variabilities due to a spectrum of computational workflows, and
  • limited resources to compare reproducibility across many hardware and software workflow solutions due to the large volume of data and time-consuming complex computations

In this study, we focus on the variability of image-based measurements that come from

  1. computational approaches (methods),
  2. implementations of methods,
  3. parameter settings,
  4. chaining methods into workflows, and
  5. stabilities of floating-point arithmetic on diverse hardware.

We approach the measurement variability by

  • introducing interoperability between algorithms,
  • enforcing automated capture of computational provenance and parameter settings, and
  • quantifying multiple sources of variabilities for several nucleus measurements from many workflow streams executed in multiple workflow configurations, on a few computational hardware platforms at multiple research institutions.

Major Accomplishments

  • Containerized methods for image segmentation: Mask Region CNN and Feature Pyramid Networks
  • Compared execution time for multiple segmentation methods using different hardware specifications and parameters.
  • Containerized a training method with transfer learning using a large dataset (COCO or ImageNet) and the desired training dataset.
  • Quantified sources of variability in image-based nucleus measurements (see the publication below).



Software pointers:

WIPP Plugins:

  • Mask R-CNN plugin - Docker image: wipp/wipp-mrcnn-inference-plugin:1.6
  • Feature Pyramid Networks - Docker image: wipp/wipp-fpn-inference-plugin:1.0

WIPP plugin registry:

Created May 28, 2021, Updated June 11, 2021