Hybrid-LLM-GNN: Integrating Large Language Models and Graph Neural Networks for Enhanced Materials Property Prediction

Published

December 18, 2024

Author(s)

Youjia Li, Vishu Gupta, Talha Killic, Kamal Choudhary, Daniel Wines, Wei-keng Liao, Alok Choudhary, Ankit Agrawal

Abstract

Graph-centric learning has attracted significant interest in material informatics. Accordingly, a family of graph-based machine learning models, primarily utilizing Graph Neural Networks (GNN), has been developed to provide accurate material properties predictions. In recent years, Large Language Models (LLM) have revolutionized existing scientific workflows that process text representations, thanks to their exceptional ability to utilize extensive common knowledge for understanding semantics. With the help of automated text representation tools, fine-tuned LLMs have demonstrated competitive prediction accuracy as standalone predictors. In this paper, we propose to integrate the insights from GNNs and LLMs to enhance both prediction accuracy and model interpretability. Inspired by the feature extraction-based transfer learning study for the GNN model, we introduce a novel framework that extracts and combines GNN and LLM embeddings to predict material properties. We evaluated the proposed framework in cross-property scenarios using 7 properties to find that the combination of MatBERT and ALIGNN outperforms the ALIGNN-based only transfer learning approach in all 7 cases with a maximum MAE reduction of 9.21 %. We conduct rationale comprehensiveness analysis through text erasure to interpret the model predictions by examining the contribution of different parts of the text representation.

Citation

Digital Discovery

Pub Type

Journals

Download Paper

https://doi.org/10.1039/D4DD00199K

Local Download

Citation

Li, Y. , Gupta, V. , Killic, T. , Choudhary, K. , Wines, D. , Liao, W. , Choudhary, A. and Agrawal, A. (2024), Hybrid-LLM-GNN: Integrating Large Language Models and Graph Neural Networks for Enhanced Materials Property Prediction, Digital Discovery, [online], https://doi.org/10.1039/D4DD00199K, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=958100 (Accessed January 17, 2025)