A reproducibility study of graph neural networks for materials property prediction

Published

April 30, 2024

Author(s)

Kangming Li, Brian DeCost, Kamal Choudhary, Jason Hattrick-Simpers

Abstract

Use of machine learning has been increasingly popular in materials science as data-driven materials discovery is becoming the new paradigm. Reproducibility of findings is paramount for promoting transparency and accountability in research and building trust in the scientific community. Here we conduct a reproducibility analysis of the work by K. Choudhary and B. Brian [npj Comput. Mater. 7, 185 (2021)], in which a new graph neural network architecture was developed with improved performance on multiple atomistic prediction tasks. We examine the reproducibility for the model performance on 17 crystal properties and for the ablation analysis of the graph neural network layers. We find that the reproduced results generally exhibit a good quantitative agreement with the initial study, despite minor disparities in model performance and training efficiency that may be resulting from factors such as hardware difference and stochasticity involved in model training and data splits. The ease of conducting these reproducibility experiments confirms the great benefits of open data and code practices to which the initial work adhered. We also discuss some further enhancements in reproducible practices such as code and data archiving and providing data identifiers used in dataset splits.

Citation

Digital Discovery

Pub Type

Journals

Download Paper

https://doi.org/10.1039/D4DD00064A

Local Download

Citation

LI, K. , DeCost, B. , Choudhary, K. and Hattrick-Simpers, J. (2024), A reproducibility study of graph neural networks for materials property prediction, Digital Discovery, [online], https://doi.org/10.1039/D4DD00064A, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=957055 (Accessed October 27, 2025)