Domain Adaptation of LLMs for Materials Science: Dataset Curation, Fine-Tuning, and Evaluation Benchmark

This thesis addresses the gap between the capabilities of general-purpose large language models (LLMs) and the specialized needs of materials science. While LLMs have shown transformative potential in many fields, their application in materials science remains limited due to the lack of domain-specific natural language datasets and evaluation benchmarks. To overcome this challenge, the thesis introduces a curated instruction-tuning dataset composed of diverse question-answer (QA) pairs drawn from various materials science sources such as textbooks, property databases, and expert forums. This dataset was used to fine-tune the LLaMA-3-8B-Instruct model, to create a materials science domain expert. However, the experiments revealed both fine-tuning datasets and evaluation limitations. These findings led to a refined, higher-quality second pipeline focused on building an evaluation benchmark rather than a full training corpus. The final benchmark includes 2,100 QA pairs across a wide range of materials science topics, with a subset validated by human experts. The thesis adopts an LLM-as-judge framework to evaluate models on this benchmark, where GPT-4o is used to compare model answers against gold responses using a rubric. We are conducting a human expert annotation to show that GPT-4o, as a judge, is reliable. Ultimately, this work contributes a complete pipeline—from dataset construction to evaluation—for developing and benchmarking domain-specific LLMs in materials science. It lays the groundwork for future efforts toward using LLMs in scientific research.

Level thesis:degree_level

Masters

Department dc:contributor.department

Computer Science

Grantor dc:publisher

Georgia Institute of Technology

Year dc:date.issued

2025

Author dc:creator

• Shen, Shiyao

Advisor dc:contributor.advisor

• Zhang, Chao

Committee members dc:contributor.committeemember

• Fung, Victor
• Ramprasad, Rampi

• Benchmark Curation
• LLM as a judge

Language dc:language.iso

en_US

Handle dc:identifier.uri

https://hdl.handle.net/1853/81505

OAI identifier oai:identifier

oai:repository.gatech.edu:1853/81505