{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/124537"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/124537","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"CrystalCoder and CrystalChat: Illuminating LLM abilities on language and code","abstract":"Submission published under a 24 month embargo labeled 'U of I Access', the embargo will last until 2026-05-01","abstract_html":"Submission published under a 24 month embargo labeled 'U of I Access', the embargo will last until 2026-05-01","abstract_has_math":false,"creators":["Tao, Tianhua"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"M.S.","degree_level":"Thesis","degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":["Peng, Hao"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2024,"date_issued":"2024-05","date_published":"2024-05","updated_at":"2026-07-22T22:25:02Z","subjects":["Natural Language Processing","Language Model"],"languages":["en","eng"],"rights":["Copyright 2024 Tianhua Tao"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"https://hdl.handle.net/2142/124537","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Peng, Hao"]},{"key":"dc:creator","label":"Author","values":["Tao, Tianhua"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2024-05","2024-04-25"]},{"key":"dc:type","label":"Dc Type","values":["text"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Thesis"]},{"key":"thesis:degree_name","label":"Degree Name","values":["M.S."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Natural Language Processing","Language Model"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en","eng"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2024 Tianhua Tao"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://hdl.handle.net/2142/124537"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Submission published under a 24 month embargo labeled 'U of I Access', the embargo will last until 2026-05-01","The student, Tianhua Tao, accepted the attached license on 2024-04-17 at 00:15.","The student, Tianhua Tao, submitted this Thesis for approval on 2024-04-17 at 00:22.","This Thesis was approved for publication on 2024-04-25 at 14:40.","DSpace SAF Submission Ingestion Package generated from Vireo submission #20454 on 2024-09-16 at 00:43:46","Large Language Models (LLMs) specializing in code generation, which are also often referred to as code LLMs, e.g., StarCoder and Code Llama, play increasingly critical roles in various software development scenarios. It is also crucial for code LLMs to possess both code generation and natural language abilities for many specific applications, such as code snippet retrieval using natural language and code usage instruction generation. The intricate interaction between acquiring language and coding skills complicates the development of strong code LLMs. In open-sourced LLMs, we observe a prevalent issue: most models are tailored to specialize in either language or code, not both. For example, Llama is proficient in natural language tasks but poor in code tasks, while Code Llama is the opposite. Furthermore, there is a lack of thorough prior studies on LLM pretraining strategies that mix code and natural language. In this work, we propose a pretraining strategy designed to enhance the integration of natural language and coding capabilities within a single LLM. Specifically, it includes three pretraining phases with appropriately adjusted code/language ratios. The resulting model, CrystalCoder, achieves remarkable capability in both domains. Specifically, it attains natural language and coding performance comparable to that of Llama 2 and Code Llama, respectively. CrystalCoder exhibits better data efficiency, using 1.4 trillion tokens compared to the more than 2 trillion tokens used by Llama 2 and Code Llama. We further fine-tuned the pretrained model with a collection of open-source datasets and delivered our instruction-following model, CrystalChat. We verify our pretraining strategy by analyzing the training process and observing consistent improvements in most benchmarks. To foster research within the community, we commit to open-sourcing every detail of the pretraining, including our training datasets, code, and 136 checkpoints throughout the training."]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["CrystalCoder and CrystalChat: Illuminating LLM abilities on language and code"]}]}],"canonical_facts":{"dc:contributor":["Peng, Hao"],"dc:creator":["Tao, Tianhua"],"dc:date":["2024-05","2024-04-25"],"dc:description":["Submission published under a 24 month embargo labeled 'U of I Access', the embargo will last until 2026-05-01","The student, Tianhua Tao, accepted the attached license on 2024-04-17 at 00:15.","The student, Tianhua Tao, submitted this Thesis for approval on 2024-04-17 at 00:22.","This Thesis was approved for publication on 2024-04-25 at 14:40.","DSpace SAF Submission Ingestion Package generated from Vireo submission #20454 on 2024-09-16 at 00:43:46","Large Language Models (LLMs) specializing in code generation, which are also often referred to as code LLMs, e.g., StarCoder and Code Llama, play increasingly critical roles in various software development scenarios. It is also crucial for code LLMs to possess both code generation and natural language abilities for many specific applications, such as code snippet retrieval using natural language and code usage instruction generation. The intricate interaction between acquiring language and coding skills complicates the development of strong code LLMs. In open-sourced LLMs, we observe a prevalent issue: most models are tailored to specialize in either language or code, not both. For example, Llama is proficient in natural language tasks but poor in code tasks, while Code Llama is the opposite. Furthermore, there is a lack of thorough prior studies on LLM pretraining strategies that mix code and natural language. In this work, we propose a pretraining strategy designed to enhance the integration of natural language and coding capabilities within a single LLM. Specifically, it includes three pretraining phases with appropriately adjusted code/language ratios. The resulting model, CrystalCoder, achieves remarkable capability in both domains. Specifically, it attains natural language and coding performance comparable to that of Llama 2 and Code Llama, respectively. CrystalCoder exhibits better data efficiency, using 1.4 trillion tokens compared to the more than 2 trillion tokens used by Llama 2 and Code Llama. We further fine-tuned the pretrained model with a collection of open-source datasets and delivered our instruction-following model, CrystalChat. We verify our pretraining strategy by analyzing the training process and observing consistent improvements in most benchmarks. To foster research within the community, we commit to open-sourcing every detail of the pretraining, including our training datasets, code, and 136 checkpoints throughout the training."],"dc:format":["application/pdf"],"dc:identifier":["https://hdl.handle.net/2142/124537"],"dc:language":["en","eng"],"dc:rights":["Copyright 2024 Tianhua Tao"],"dc:subject":["Natural Language Processing","Language Model"],"dc:title":["CrystalCoder and CrystalChat: Illuminating LLM abilities on language and code"],"dc:type":["text"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_level":["Thesis"],"thesis:degree_name":["M.S."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:25:02Z"}