Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

JARVIS-Leaderboard: A Large Scale Benchmark of Materials Design Methods

Published

Author(s)

Kamal Choudhary, Daniel Wines, Kevin Garrity, aldo romero, Jaron Krogel, Kayahan Saritas, Panchapakesan Ganesh, Paul Kent, Pascal Friederich, Vishu Gupta, Ankit Agrawal, Pratyush Tiwary, ichiro takeuchi, Robert Wexler, Arun Kumar Mannodi-Kanakkithodi, Avanish Mishra, Kangming Li, Adam Biacchi, Francesca Tavazza, Ben Blaiszik, Jason Hattrick-Simpers, Maureen E. Williams

Abstract

Reproducibility and validation are major hurdles for scientific development across many fields. Materials science in particular encompasses a variety of experimental and theoretical approaches that require careful benchmarking. Leaderboard efforts have been developed previously to mitigate these issues, however, a comprehensive comparison and benchmarking on an integrated platform with multiple data-modalities with both perfect and defect materials data is still lacking. This work introduces the JARVIS-Leaderboard, an open-source and community-driven platform that facilitates benchmarking and enhances reproducibility. The platform allows users to set up benchmarks with custom tasks and enables contributions in the form of dataset, code, and meta-data submissions. We cover the following materials design categories: Artificial Intelligence (AI), Electronic Structure (ES), Force-fields (FF), Quantum Computation (QC), and Experiments (EXP). For AI, we cover several types of input data, including atomic structures, atomistic images, spectra, and text. For ES, we consider multiple ES approaches, software packages, pseudopotentials, materials, and properties, comparing results to experiment. For FF, we compare multiple approaches for material property predictions. For QC, we benchmark Hamiltonian simulations using various quantum algorithms and circuits. Finally, for experiments, we use the round-robin approach to establish benchmarks. Currently, there are 1008 contributions to 225 benchmarks using over 100 different methods, and the leaderboard is continuously expanding. The JARVIS-Leaderboard is available at the website: \urlhttps://pages.nist.gov/jarvis_leaderboard/}
Citation
npj Computational Materials

Keywords

JARVIS, Reproducibility, benchmarking

Citation

Choudhary, K. , Wines, D. , Garrity, K. , Romero, A. , Krogel, J. , Saritas, K. , Ganesh, P. , Kent, P. , Friederich, P. , Gupta, V. , Agrawal, A. , Tiwary, P. , Takeuchi, I. , Wexler, R. , Mannodi-Kanakkithodi, A. , Mishra, A. , LI, K. , Biacchi, A. , Tavazza, F. , Blaiszik, B. , Hattrick-Simpers, J. and Williams, M. (2024), JARVIS-Leaderboard: A Large Scale Benchmark of Materials Design Methods, npj Computational Materials, [online], https://doi.org/10.1038/s41524-024-01259-w, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=936953 (Accessed June 13, 2024)

Issues

If you have any questions about this publication or are having problems accessing it, please contact reflib@nist.gov.

Created May 7, 2024, Updated May 16, 2024