Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

PROJECTS/PROGRAMS

Reproducibility of Imaging Analyses Applied to Nucleus Image Quantification

Summary

Our motivation for addressing the variability problem in image-based measurements comes from increasing reports of irreproducibility in the Artificial Intelligence/Machine learning (AI/ML) field. Most recently, two new analyses [1, 2] put the spotlight on machine learning in health research, where lack of reproducibility and poor quality could risk harm to patients and/or lower the quality of care a patient receives. There is a need to quantify and minimize measurement uncertainties from various computational sources while leveraging all cutting-edge AI/ML approaches to image-based drug discoveries.

[1] M. B. A. McDermott, S. Wang, N. Marinsek, R. Ranganath, L. Foschini, and M. Ghassemi, “Reproducibility in machine learning for health research: Still a ways to go,” Sci. Transl. Med., vol. 13, no. 586, Mar. 2021, DOI: 10.1126/scitranslmed.abb1655.

[2] M. Roberts et al., “Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans,” Nat. Mach. Intell., vol. 3, no. 3, pp. 199--217, Mar. 2021, DOI: 10.1038/s42256-021-00307-0.

Description

The goal of our effort is to improve reproducibility of image-based measurements and quantify the measurement uncertainty while leveraging all cutting-edge AI/ML approaches to image-based drug discoveries. In our work, we selected measurements derived from fluorescently-labeled nucleus images over samples with a variety of drug treatments and imaging variations. Such image-based measurements require

nucleus segmentation from background,
labeling each nucleus with a unique label, and
computing intensity and shape characteristics per unique label of a nucleus.

The challenges of delivering reproducible nucleus measurements lie in a lack of

interoperability of cutting-edge approaches and their implementations,
automatic capturing of computational provenance about input and intermediate datasets, as well as about parameter settings, for sharing across teams and institutions,
understanding of variabilities due to a spectrum of computational workflows, and
limited resources to compare reproducibility across many hardware and software workflow solutions due to the large volume of data and time-consuming complex computations

In this study, we focus on the variability of image-based measurements that come from

computational approaches (methods),
implementations of methods,
parameter settings,
chaining methods into workflows, and
stabilities of floating-point arithmetic on diverse hardware.

We approach the measurement variability by

introducing interoperability between algorithms,
enforcing automated capture of computational provenance and parameter settings, and
quantifying multiple sources of variabilities for several nucleus measurements from many workflow streams executed in multiple workflow configurations, on a few computational hardware platforms at multiple research institutions.

Major Accomplishments

Containerized methods for image segmentation: Mask Region CNN and Feature Pyramid Networks
Compared execution time for multiple segmentation methods using different hardware specifications and parameters.
Containerized a training method with transfer learning using a large dataset (COCO or ImageNet) and the desired training dataset.
Quantified sources of variability in image-based nucleus measurements (see the publication below).