# Exploring the Expertise of Large Language Models in Materials Science and Metallurgical Engineering

https://mdr.nims.go.jp/datasets/25df19d4-bd2e-4da3-84af-d095a35fdc8f

## File

- [d4dd00319e.pdf](https://mdr.nims.go.jp/filesets/a89028d2-af55-4305-9b93-016bd3f9daa7/download) ([Detail](https://mdr.nims.go.jp/filesets/a89028d2-af55-4305-9b93-016bd3f9daa7.md))
- [d4dd00319e1.pdf](https://mdr.nims.go.jp/filesets/f7118ebd-72c8-4aa7-94e3-a62c0d8ba766/download) ([Detail](https://mdr.nims.go.jp/filesets/f7118ebd-72c8-4aa7-94e3-a62c0d8ba766.md))

## Id

25df19d4-bd2e-4da3-84af-d095a35fdc8f

## Local identifier



## Visibility

open_to_public

## State

published

## Created at

2025-01-20T08:02:02.989456Z

## Updated at

2025-01-21T03:30:37.295122Z

## Published at

2025-01-21T03:30:37.384294Z

## Doi



## First published url

https://doi.org/10.1039/d4dd00319e

## Date published

2025-01-20

## Recorded date published

2025-2-12

## Resource type

journal_article

## Manuscript type

vor

## Collection



## Title

- title: Exploring the Expertise of Large Language Models in Materials Science and
    Metallurgical Engineering
  title_type: original
  lang: en

## Description

- description: The integration of artificial intelligence into various domains is
    rapidly increasing, with Large Language Models (LLMs) becoming more prevalent
    in numerous applications. This work is include in an overall project which aims
    to train an LLM specifically in the field of materials science. To assess the
    impact of this specialized training, it is essential to establish the baseline
    performance of existing LLMs in materials science. In this study, we evaluated
    15 different LLMs using the MaScQA question answering (Q&A) benchmark. This benchmark
    comprises questions from the Graduate Aptitude Test in Engineering (GATE), tailored
    to test models' capabilities in answering questions related to the materials science
    and metallurgical engineering. Our results indicate that closed-source LLMs, such
    as Claude-3.5-Sonnet and GPT-4o, perform the best with an overall accuracy of
    84%, when the open-source models, Llama3-70b and Phi3-14b, top at 56 and 43%,
    respectively. These findings provide a baseline for the raw capabilities of LLMs
    on Q&A tasks applied to materials science, and emphasise the substantial improvement
    that could be brought to open-source models via prompt engineering and fine-tuning
    strategies. We anticipate that this work could push the adoption of LLMs as valuable
    assistants in materials science, demonstrating their utilities in this specialised
    domain and related sub-domains.
  description_type: abstract
  lang: eng

## Creator

- name: Christophe Bajan
  role: author
  orcid: https://orcid.org/0009-0008-1433-9618
  organization: National Institute for Materials Science
  department: Center for Basic Research on Materials/Data-driven Materials Research
    Field/Data-driven Materials Design Group
- name: Guillaume Lambard
  role: author
  orcid: https://orcid.org/0000-0003-0275-4079
  organization: National Institute for Materials Science
  department: Center for Basic Research on Materials/Data-driven Materials Research
    Field/Data-driven Materials Design Group

## Contact agent



## Publisher

organization: Royal Society of Chemistry

## Managing organization



## Keyword

- subject: Large language models
  schema: not_defined
- subject: Materials science
  schema: not_defined
- subject: Metallurgical Engineering
  schema: not_defined

## Rights

- identifier: https://creativecommons.org/licenses/by/4.0/

## Other identifier(s)



## Data origin

- data_origin_type: other

## Embargo



## Journal

- title: Digital Discovery
  issn: 2635098X

## Conference



## Related item



## Funding



## Instrument



## Instrument operator



## Instrument managing organization



## Measurement method



## Specimen



## Chemical composition



## Structure for specimen



## Structural feature for specimen



## Specific property for specimen



## Process for specimen treatment



## Computational method



## Energy level/transition state



## Software



## Custom property



## Fileset

- id: a89028d2-af55-4305-9b93-016bd3f9daa7
  filename: d4dd00319e.pdf
  content_type: application/pdf
  size: 1308521
  md5: 99e13a29282661b7062bc5cba8a5f943
- id: f7118ebd-72c8-4aa7-94e3-a62c0d8ba766
  filename: d4dd00319e1.pdf
  content_type: application/pdf
  size: 482178
  md5: '098191b5df7eded6e44fc5218c21c9c4'

## Thumbnail

fileset_id: a89028d2-af55-4305-9b93-016bd3f9daa7
filename: d4dd00319e.pdf