QLoRaX: Heuristic-Guided Fine-Tuning of LLaMA-2 for Domain Adaptation in Entrepreneurship

<p>This thesis presents a study on the fine-tuning of large language models (LLMs) for domain-specific applications using limited data. We fine-tuned the LLaMA-2 (7B) model on a curated entrepreneurial dataset containing 3,545 human-written question-answer pairs, of which 3,095 were used for training and 450 were reserved for evaluation. A complete fine-tuning and evaluation pipeline was developed, which included clustering human-written answers, generating centroid-based summaries for each cluster, and evaluating the model's generated responses through cosine similarity.</p> <p>Training was carried out over five epochs, with model performance evaluated after each epoch. The fine-tuned model demonstrated strong semantic alignment with human-written content, producing coherent, detailed, and contextually appropriate answers tailored to entrepreneurial questions. Through structured clustering and summary-based evaluation, we established a reliable framework for assessing model quality in specialized domains. We evaluated model performance based on training loss and time. In the first experiment (LoRA parameters only), the fastest epoch took 402.7 seconds with a loss of 1.7690; the lowest loss was 1.7513 in 407 seconds. In the second experiment (LoRA + LLaMA-2), the shortest epoch was 399.8 seconds with a loss of 1.9587; the lowest loss was 1.7576 in 404 seconds. After fine-tuning, the model achieved a cosine similarity of 0.4744 on the original 25 questions. For five paraphrased questions, scores ranged from 0.4949 to 0.6606. For three general questions, the scores were 0.3409, 0.5294, and 0.401.</p> <p>This work successfully demonstrates that large language models can be effectively adapted to niche areas even when working with modest-sized datasets. The results highlight the potential of structured fine-tuning techniques combined with efficient evaluation strategies to achieve high-quality domain adaptation, contributing to the broader goal of making large models more accessible and specialized for practical real-world applications</p>

Name thesis:degree_name

Computer Science - Applied Computing Track

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

TSYS School of Computer Science

Year

2025

Author dc:creator

• Saha, Gaurob

Contributors dc:contributor

• Dr. Rania Hodhod
• Dr. Lydia Ray
• Dr. Riduan Abid

• LLMs
• Large Language Models
• Computer Sciences
• Databases and Information Systems

Language dc:language

English

Repository record dc:identifier

https://csuepress.columbusstate.edu/theses_dissertations/575

OAI identifier oai:identifier

oai:csuepress.columbusstate.edu:theses_dissertations-1577