{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/124128"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/124128","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Control software system","abstract":"Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2024-09-16 without embargo terms","abstract_html":"Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2024-09-16 without embargo terms","abstract_has_math":false,"creators":["Tao, Heyi"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"M.S.","degree_level":"Thesis","degree_discipline":"Electrical & Computer Engr","degree_department":null,"school":null,"contributors":["Hoiem, Derek W"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2024,"date_issued":"2024-05","date_published":"2024-05","updated_at":"2026-07-22T22:25:00Z","subjects":["Computer Vision","Artificial Intelligence","Machine Learning","Natural Language Processing"],"languages":["en","eng"],"rights":["Copyright 2023 Heyi Tao"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"https://hdl.handle.net/2142/124128","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Hoiem, Derek W"]},{"key":"dc:creator","label":"Author","values":["Tao, Heyi"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2024-05","2024-02-02"]},{"key":"dc:type","label":"Dc Type","values":["text"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Electrical & Computer Engr"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Thesis"]},{"key":"thesis:degree_name","label":"Degree Name","values":["M.S."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Computer Vision","Artificial Intelligence","Machine Learning","Natural Language Processing"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en","eng"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2023 Heyi Tao"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://hdl.handle.net/2142/124128"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2024-09-16 without embargo terms","The student, Heyi Tao, accepted the attached license on 2023-12-08 at 13:26.","The student, Heyi Tao, submitted this Thesis for approval on 2023-12-08 at 13:36.","This Thesis was approved for publication on 2024-02-02 at 10:51.","DSpace SAF Submission Ingestion Package generated from Vireo submission #20184 on 2024-09-16 at 00:32:57","A primary objective in AI development is to empower AI agents to effectively interact with their environments and proficiently perform various tasks with good generalization ability. Web-based computer software programs serve as excellent examples due to their versatility, encompassing a wide range of tasks, each with numerous variations. Ideally, designed system should be able to understand how to correctly perform those tasks based on their interfaces as well as natural language descriptions given pre-defined actions without having large amount of demonstration examples or training data. Previous approaches, such as reinforcement learning (RL) or imitation learning (IL), are inefficient to train and often task-specific. In this paper, we explores the development of a system that leverages large language models (LLMs) to automatically execute web-based software tasks through actions such as clicking, entering text, and scrolling. As the observation to the software interfaces, we utilize the Document Object Model (DOM) elements. Our system only generates and executes one action at a time based on the current observations of the interface. We also provide either one human manually annotated example, or an automatically generated example based on a successful zero-shot trial by letting LLM self-exploring the correct programs to correctly solve the tasks. Our designed system has been evaluated on the MiniWob++ benchmark. By having only one in-context learning example, our system achieves comparable performance than other methods that have been used many training data or demonstrations and trials."]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Control software system"]}]}],"canonical_facts":{"dc:contributor":["Hoiem, Derek W"],"dc:creator":["Tao, Heyi"],"dc:date":["2024-05","2024-02-02"],"dc:description":["Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2024-09-16 without embargo terms","The student, Heyi Tao, accepted the attached license on 2023-12-08 at 13:26.","The student, Heyi Tao, submitted this Thesis for approval on 2023-12-08 at 13:36.","This Thesis was approved for publication on 2024-02-02 at 10:51.","DSpace SAF Submission Ingestion Package generated from Vireo submission #20184 on 2024-09-16 at 00:32:57","A primary objective in AI development is to empower AI agents to effectively interact with their environments and proficiently perform various tasks with good generalization ability. Web-based computer software programs serve as excellent examples due to their versatility, encompassing a wide range of tasks, each with numerous variations. Ideally, designed system should be able to understand how to correctly perform those tasks based on their interfaces as well as natural language descriptions given pre-defined actions without having large amount of demonstration examples or training data. Previous approaches, such as reinforcement learning (RL) or imitation learning (IL), are inefficient to train and often task-specific. In this paper, we explores the development of a system that leverages large language models (LLMs) to automatically execute web-based software tasks through actions such as clicking, entering text, and scrolling. As the observation to the software interfaces, we utilize the Document Object Model (DOM) elements. Our system only generates and executes one action at a time based on the current observations of the interface. We also provide either one human manually annotated example, or an automatically generated example based on a successful zero-shot trial by letting LLM self-exploring the correct programs to correctly solve the tasks. Our designed system has been evaluated on the MiniWob++ benchmark. By having only one in-context learning example, our system achieves comparable performance than other methods that have been used many training data or demonstrations and trials."],"dc:format":["application/pdf"],"dc:identifier":["https://hdl.handle.net/2142/124128"],"dc:language":["en","eng"],"dc:rights":["Copyright 2023 Heyi Tao"],"dc:subject":["Computer Vision","Artificial Intelligence","Machine Learning","Natural Language Processing"],"dc:title":["Control software system"],"dc:type":["text"],"thesis:degree_discipline":["Electrical & Computer Engr"],"thesis:degree_level":["Thesis"],"thesis:degree_name":["M.S."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:25:00Z"}