{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/127252"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/127252","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Evaluating long context code understanding of large language models","abstract":"Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2025-03-28 without embargo terms","abstract_html":"Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2025-03-28 without embargo terms","abstract_has_math":false,"creators":["Tian, Jia Le"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"M.S.","degree_level":"Thesis","degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":["Zhang, Lingming"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2024,"date_issued":"2024-12-04","date_published":"2024-12-04","updated_at":"2026-07-22T22:25:03Z","subjects":["Large Language Models","Language And Computation","Software Engineering","Machine Learning"],"languages":["en","eng"],"rights":["Copyright 2024 Jia Le Tian"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"https://hdl.handle.net/2142/127252","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Zhang, Lingming"]},{"key":"dc:creator","label":"Author","values":["Tian, Jia Le"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2024-12-04","2024-12"]},{"key":"dc:type","label":"Dc Type","values":["text","Thesis"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Thesis"]},{"key":"thesis:degree_name","label":"Degree Name","values":["M.S."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Large Language Models","Language And Computation","Software Engineering","Machine Learning"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en","eng"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2024 Jia Le Tian"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://hdl.handle.net/2142/127252"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2025-03-28 without embargo terms","The student, Jia Le Tian, accepted the attached license on 2024-12-02 at 17:07.","The student, Jia Le Tian, submitted this Thesis for approval on 2024-12-02 at 17:08.","This Thesis was approved for publication on 2024-12-04 at 14:03.","DSpace SAF Submission Ingestion Package generated from Vireo submission #21450 on 2025-03-28 at 14:27:45","The rise and adoption of Large Language Models has brought on a new era of Software Engineering, where LLMs are at the center of the development cycle. Within this rapid growth, the need to quantify code understanding of models is ever more critical. Code understanding is a foundational skill relevant to many downstream coding tasks and is a necessary ability for LLMs to possess in order to push their coding potential. To benchmark the code understanding capabilities of Large Language Models, this work introduces the Searching Needle Function (SNF) benchmark, a set of 500 code retrieval problems spanning across five programming languages (Python, Java, TypeScript, Rust and C++) designed to test the code understanding capabilities of LLMs. Inspired by the famous Needle in a HayStack task, the Searching Needle Function task emphasizes the long context code understanding capabilities of models through retrieval within a long input. Unlike the NIH task, however, the SNF task is realistic and is composed entirely of popular real-world open-source code repositories for each language. The SNF task involves retrieving a needle function in its entirety given only (i) a natural language description and (ii) a long context containing the needle function. This work details the construction process and evaluation criteria for the Searching Needle Function task. This work also evaluates the benchmark against 33 state-of-the-art open and closed-source models. Through this extensive evaluation, this work constructs a comprehensive ranking of all current state-of-the-art models. More specifically, the evaluation found that (i) closed-sourced models vastly outperform open-sourced models at long context code understanding (ii) TypeScript and Java are the best languages for current LLMs to understand and (iii) removing comments from the input code context can improve the overall performance of models."]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Evaluating long context code understanding of large language models"]}]}],"canonical_facts":{"dc:contributor":["Zhang, Lingming"],"dc:creator":["Tian, Jia Le"],"dc:date":["2024-12-04","2024-12"],"dc:description":["Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2025-03-28 without embargo terms","The student, Jia Le Tian, accepted the attached license on 2024-12-02 at 17:07.","The student, Jia Le Tian, submitted this Thesis for approval on 2024-12-02 at 17:08.","This Thesis was approved for publication on 2024-12-04 at 14:03.","DSpace SAF Submission Ingestion Package generated from Vireo submission #21450 on 2025-03-28 at 14:27:45","The rise and adoption of Large Language Models has brought on a new era of Software Engineering, where LLMs are at the center of the development cycle. Within this rapid growth, the need to quantify code understanding of models is ever more critical. Code understanding is a foundational skill relevant to many downstream coding tasks and is a necessary ability for LLMs to possess in order to push their coding potential. To benchmark the code understanding capabilities of Large Language Models, this work introduces the Searching Needle Function (SNF) benchmark, a set of 500 code retrieval problems spanning across five programming languages (Python, Java, TypeScript, Rust and C++) designed to test the code understanding capabilities of LLMs. Inspired by the famous Needle in a HayStack task, the Searching Needle Function task emphasizes the long context code understanding capabilities of models through retrieval within a long input. Unlike the NIH task, however, the SNF task is realistic and is composed entirely of popular real-world open-source code repositories for each language. The SNF task involves retrieving a needle function in its entirety given only (i) a natural language description and (ii) a long context containing the needle function. This work details the construction process and evaluation criteria for the Searching Needle Function task. This work also evaluates the benchmark against 33 state-of-the-art open and closed-source models. Through this extensive evaluation, this work constructs a comprehensive ranking of all current state-of-the-art models. More specifically, the evaluation found that (i) closed-sourced models vastly outperform open-sourced models at long context code understanding (ii) TypeScript and Java are the best languages for current LLMs to understand and (iii) removing comments from the input code context can improve the overall performance of models."],"dc:format":["application/pdf"],"dc:identifier":["https://hdl.handle.net/2142/127252"],"dc:language":["en","eng"],"dc:rights":["Copyright 2024 Jia Le Tian"],"dc:subject":["Large Language Models","Language And Computation","Software Engineering","Machine Learning"],"dc:title":["Evaluating long context code understanding of large language models"],"dc:type":["text","Thesis"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_level":["Thesis"],"thesis:degree_name":["M.S."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:25:03Z"}