Performance Metrics for Proteomics

In collaboration with the National Cancer Institute's Clinical Proteomic Technologies for Cancer (CPTC), NIST researchers are developing specific and quantitative metrics, implemented in a software pipeline, for evaluating sources of technical variability present in mass spectrometry-based proteomics experiments. The use of these metrics by community researchers is helping to optimize these platforms and reduce technical variability which can undermine the search for new diagnostic and prognostic protein "biomarkers."

Intended Impact

Proteomics experiments designed to identify protein markers specific to a particular disease or disease state rely on repeatable identification of proteins within a control set. Repeatability provides statistical assurance that clear differences between two sets (e.g. sick vs. healthy) reflect true biological differences.  The degree to which technical variability thwarts this process is an underappreciated problem in the field. Monitoring key steps in these experimental workflows using metrics, provides researchers the necessary tools to evaluate variability and determine the degree to which differences are due to biological changes. Their use will generally improve confidence in this area and speed the search for new disease markers.

Objective

Mass spectrometry is a powerful technology which has been successfully adapted for studying the protein component of tissues and body fluids (i.e. proteomics). Technical variability is high in this newly emerging field because of the complexity of the experimental workflows. Understanding and measuring this variability is essential for firmly establishing proteomics as a competitive technology for tomorrow's health care in the United States.

Goals

The goals of this project are to identify and measure experimental variability in proteomics experiments as it affects:

• liquid chromatography
• mass spectrometric sampling of ions
• the identification of peptides by interpretation of tandem mass spectra

By systematically monitoring changes across these key areas, outlier data can be easily identified, problems diagnosed and corrective measures made.

Research Activities and Technical Approach

The NCI's Clinical Proteomic Technologies for Cancer (CPTC) have identified technical variability as a major roadblock in the search for protein markers of cancer and the development of the next-generation of cancer detection and treatment technologies. Through an interagency agreement with the NCI, NIST researchers have developed a panel of 50+ metrics, which are computed from raw mass spectrometry data and peptide identifications by spectrum library searching, for evaluating reproducibility.

These metrics have been successfully used in several interlaboratory studies as part of NCI's Clinical Proteomics Technology Assessment for Cancer (CPTAC) program as a means to study reproducibility within and between expert laboratories. These studies have involved analysis of two complex reference samples (i.e. a 20 human protein mix and yeast) by several sites affiliated with each of the 5 CPTAC awardees. The data files were transmitted and analyzed by NIST researchers for evaluation of reproducibility across each of the categories listed above. Results from these analyses, as well as from 'in-house' studies of variability, have been used to document reproducibility and provide specific feedback for reducing variability and optimizing performance.

These metrics have been implemented in a software pipeline available for download. NIST researchers are working with several universities and industrial partners to further develop these as community tools.

Publications:

Stein, S.E., ASMS 2008, "Measuring Variability in Shotgun Proteomics Experiments".

Rudnick, P.A., Clauser, K.R., Kilpatrick, L.E., Tchekhovskoi, D.V., Neta, P., Blonder, N., Billheimer, D.D., Blackman, R.K., Bunk, D.M., Cardasis, H.L., Ham, A.J., Jaffe, J.D., Kinsinger, C.R., Mesri, M., Neubert, T.A., Schilling, B., Tabb, D.L., Tegeler, T.J., Vega-Montoto, L., Variyath, A.M., Wang, M., Wang, P., Whiteaker, J.R., Zimmerman, L.J., Carr, S.A., Fisher, S.J., Gibson, B.W., Paulovich, A.G., Regnier, F.E., Rodriguez, H., Spiegelman, C., Tempst, P., Liebler, D.C., Stein, S.E., Performance Metrics for Liquid Chromatography-Tandem Mass Spectrometry Systems in Proteomics Analyses, Mol. Cell Proteomics, 2010 Feb., 9(2), pp. 225-241.

• A panel of performance metrics for proteomics has been developed for analyzing some MS vendor raw files
  
• These metrics have been successfully used to evaluate variability in large interlaboratory studies
  
• Evaluation of the software is underway at leading institutions