Download all files (1.36MB)

コレクション

引用

説明:

The integration of artificial intelligence into various domains is rapidly increasing, with Large Language Models (LLMs) becoming more prevalent in numerous applications. This work is include in an overall project which aims to train an LLM specifically in the field of materials science. To assess the impact of this specialized training, it is essential to establish the baseline performance of existing LLMs in materials science. In this study, we evaluated 15 different LLMs using the MaScQA question answering (Q&A) benchmark. This benchmark comprises questions from the Graduate Aptitude Test in Engineering (GATE), tailored to test models' capabilities in answering questions related to the materials science and metallurgical engineering. Our results indicate that closed-source LLMs, such as Claude-3.5-Sonnet and GPT-4o, perform the best with an overall accuracy of 84%, when the open-source models, Llama3-70b and Phi3-14b, top at 56 and 43%, respectively. These findings provide a baseline for the raw capabilities of LLMs on Q&A tasks applied to materials science, and emphasise the substantial improvement that could be brought to open-source models via prompt engineering and fine-tuning strategies. We anticipate that this work could push the adoption of LLMs as valuable assistants in materials science, demonstrating their utilities in this specialised domain and related sub-domains.

権利情報:

• Creative Commons BY Attribution 4.0 International
  

キーワード: Large language models, Materials science, Metallurgical Engineering

刊行年月日: 2025-01-20

出版者: Royal Society of Chemistry

掲載誌:

• Digital Discovery (ISSN: 2635098X)

研究助成金:

原稿種別: 出版者版 (Version of record)

MDR DOI:

公開URL: https://doi.org/10.1039/d4dd00319e

関連資料:

その他の識別子:

連絡先:

更新時刻: 2025-01-21 12:30:37 +0900

MDRでの公開時刻: 2025-01-21 12:30:37 +0900