{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/129584"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/129584","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Efficient LLM training and inference with contextual sparsity","abstract":"Submission published under a 24 month embargo labeled 'U of I Access', the embargo will last until 2027-05-01","abstract_html":"Submission published under a 24 month embargo labeled 'U of I Access', the embargo will last until 2027-05-01","abstract_has_math":false,"creators":["Ge, Suyu"],"institution":"University of Illinois Urbana-Champaign","degree_name":"Ph.D.","degree_level":"Dissertation","degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":["Han, Jiawei","Peng, Hao","Hakkani-Tür, Dilek","Gao, Jianfeng"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2025,"date_issued":"2025-04-25","date_published":"2025-04-25","updated_at":"2026-07-22T22:25:05Z","subjects":["Large Language Model Efficiency"],"languages":["en","eng"],"rights":["Copyright 2025 Suyu Ge"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"https://hdl.handle.net/2142/129584","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Han, Jiawei","Peng, Hao","Hakkani-Tür, Dilek","Gao, Jianfeng"]},{"key":"dc:creator","label":"Author","values":["Ge, Suyu"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2025-04-25","2025-05"]},{"key":"dc:type","label":"Dc Type","values":["text","Thesis"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Dissertation"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Ph.D."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Large Language Model Efficiency"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en","eng"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2025 Suyu Ge"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://hdl.handle.net/2142/129584"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Submission published under a 24 month embargo labeled 'U of I Access', the embargo will last until 2027-05-01","The student, Suyu Ge, accepted the attached license on 2025-04-25 at 13:25.","The student, Suyu Ge, submitted this Dissertation for approval on 2025-04-25 at 13:45.","This Dissertation was approved for publication on 2025-04-25 at 15:32.","DSpace SAF Submission Ingestion Package generated from Vireo submission #21986 on 2025-10-19 at 19:16:20","Large Language Models (LLMs) have demonstrated impressive capabilities across a wide range of natural language tasks. However, their computational demands present significant challenges, particularly regarding memory consumption and processing speed for long contexts. This thesis addresses these challenges through three interconnected research directions that collectively form a comprehensive framework for efficient transformer-based models: FastGen: An adaptive KV cache compression technique that analyzes intrinsic attention structures and dynamically retains only essential key-value pairs during inference. By recognizing that different attention heads exhibit distinct patterns (e.g., focusing on local context, special tokens, or broadly attending to all tokens), FastGen reduces KV cache memory consumption by up to 40\\% with negligible impact on generation quality. LongGen: A hybrid architecture that enables efficient long-context processing through partial contexts. This approach finetunes pretrained LLMs into an efficient architecture during context-length extension, integrating sparse attention in strategic layers while maintaining full attention where needed. LongGen achieves a 36\\% reduction in training time and 62\\% reduction in KV cache memory while maintaining strong performance on challenging long-context tasks, including perfect accuracy on needle-in-a-haystack retrieval at 128K tokens. S2-Attention: A hardware-aware sparse attention kernel that significantly accelerates both training and inference. By implementing context sharding across attention heads with careful consideration of GPU memory access patterns, S2-Attention achieves up to 15.9× speedup in attention computation and 4.5× end-to-end inference acceleration compared to dense attention baselines. Our comprehensive evaluations across different model scales (1.3B to 70B parameters) and benchmarks demonstrate that these approaches not only improve computational efficiency but also maintain or enhance model performance on challenging tasks. Together, these contributions advance the state of the art in efficient LLM deployment, enabling better utilization of computational resources and expanding the practical applications of these powerful models in resource-constrained environments and long-context scenarios."]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Efficient LLM training and inference with contextual sparsity"]}]}],"canonical_facts":{"dc:contributor":["Han, Jiawei","Peng, Hao","Hakkani-Tür, Dilek","Gao, Jianfeng"],"dc:creator":["Ge, Suyu"],"dc:date":["2025-04-25","2025-05"],"dc:description":["Submission published under a 24 month embargo labeled 'U of I Access', the embargo will last until 2027-05-01","The student, Suyu Ge, accepted the attached license on 2025-04-25 at 13:25.","The student, Suyu Ge, submitted this Dissertation for approval on 2025-04-25 at 13:45.","This Dissertation was approved for publication on 2025-04-25 at 15:32.","DSpace SAF Submission Ingestion Package generated from Vireo submission #21986 on 2025-10-19 at 19:16:20","Large Language Models (LLMs) have demonstrated impressive capabilities across a wide range of natural language tasks. However, their computational demands present significant challenges, particularly regarding memory consumption and processing speed for long contexts. This thesis addresses these challenges through three interconnected research directions that collectively form a comprehensive framework for efficient transformer-based models: FastGen: An adaptive KV cache compression technique that analyzes intrinsic attention structures and dynamically retains only essential key-value pairs during inference. By recognizing that different attention heads exhibit distinct patterns (e.g., focusing on local context, special tokens, or broadly attending to all tokens), FastGen reduces KV cache memory consumption by up to 40\\% with negligible impact on generation quality. LongGen: A hybrid architecture that enables efficient long-context processing through partial contexts. This approach finetunes pretrained LLMs into an efficient architecture during context-length extension, integrating sparse attention in strategic layers while maintaining full attention where needed. LongGen achieves a 36\\% reduction in training time and 62\\% reduction in KV cache memory while maintaining strong performance on challenging long-context tasks, including perfect accuracy on needle-in-a-haystack retrieval at 128K tokens. S2-Attention: A hardware-aware sparse attention kernel that significantly accelerates both training and inference. By implementing context sharding across attention heads with careful consideration of GPU memory access patterns, S2-Attention achieves up to 15.9× speedup in attention computation and 4.5× end-to-end inference acceleration compared to dense attention baselines. Our comprehensive evaluations across different model scales (1.3B to 70B parameters) and benchmarks demonstrate that these approaches not only improve computational efficiency but also maintain or enhance model performance on challenging tasks. Together, these contributions advance the state of the art in efficient LLM deployment, enabling better utilization of computational resources and expanding the practical applications of these powerful models in resource-constrained environments and long-context scenarios."],"dc:format":["application/pdf"],"dc:identifier":["https://hdl.handle.net/2142/129584"],"dc:language":["en","eng"],"dc:rights":["Copyright 2025 Suyu Ge"],"dc:subject":["Large Language Model Efficiency"],"dc:title":["Efficient LLM training and inference with contextual sparsity"],"dc:type":["text","Thesis"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_level":["Dissertation"],"thesis:degree_name":["Ph.D."],"thesis:institution_name":["University of Illinois Urbana-Champaign"]},"updated_at":"2026-07-22T22:25:05Z"}