{"id":{"repo_id":"mit","oai_identifier":"oai:dspace.mit.edu:1721.1/151317"},"canonical_url":"https://search.dev.ndltd.org/etd/mit/oai:dspace.mit.edu:1721.1/151317","repository":{"repo_id":"mit","name":"MIT","base_url":"https://dspace.mit.edu/oai/request"},"display":{"title":"NDF-Based API for Human-assisted Language Planning (HaLP)","abstract":"Recent works have show the promise of LLMs for generalizable task planning. Challenges in integrating LLM for high-level planning include outputting infeasible or sub-optimal plans, but the potentials include cultural commonsense to reason about high-level tasks on par with a human. Related works will generate free-form text that may contain actions inaccessible to the robot or over constrain the planner by providing it a static set of possible actions to select from and output mediocre plans. Humans can also decide when a task is infeasible due to limitations in the action space, and try to propose alternative plans. We show that LMs are also able to. We present an LLM-planner with the ability to request online learning of skills to output and execute optimal tabletop manipulation plans, even when the initial set of robot skills is insufficient. We build a fullstack system and deploy our method in simulation and hardware to demonstrate the capabilities of the planner and the preference for these plans over others in our ablation experiments. To support the learning of new skills, we present a low-level control API conditioned on natural language using Neural Descriptor Fields (NDFs) for out-of-plane category level manipulation that is SE(3)-equivariant and highly data-efficient to enable online learning.","abstract_html":"Recent works have show the promise of LLMs for generalizable task planning. Challenges in integrating LLM for high-level planning include outputting infeasible or sub-optimal plans, but the potentials include cultural commonsense to reason about high-level tasks on par with a human. Related works will generate free-form text that may contain actions inaccessible to the robot or over constrain the planner by providing it a static set of possible actions to select from and output mediocre plans. Humans can also decide when a task is infeasible due to limitations in the action space, and try to propose alternative plans. We show that LMs are also able to. We present an LLM-planner with the ability to request online learning of skills to output and execute optimal tabletop manipulation plans, even when the initial set of robot skills is insufficient. We build a fullstack system and deploy our method in simulation and hardware to demonstrate the capabilities of the planner and the preference for these plans over others in our ablation experiments. To support the learning of new skills, we present a low-level control API conditioned on natural language using Neural Descriptor Fields (NDFs) for out-of-plane category level manipulation that is SE(3)-equivariant and highly data-efficient to enable online learning.","abstract_has_math":false,"creators":["Fong, Alisha"],"institution":"Massachusetts Institute of Technology","degree_name":"Master","degree_level":null,"degree_discipline":null,"degree_department":"Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science","school":null,"contributors":[],"advisors":["Agrawal, Pulkit"],"committee_chairs":[],"committee_members":[],"year":2023,"date_issued":"2023-06","date_published":"2023-06","updated_at":"2026-07-22T22:21:00Z","subjects":[],"languages":[],"rights":["In Copyright - Educational Use Permitted","Copyright retained by author(s)"],"rights_urls":["https://rightsstatements.org/page/InC-EDU/1.0/"],"identifier_entries":[]},"links":{"outbound_url":"https://hdl.handle.net/1721.1/151317","outbound_label":"Handle","outbound_source":"dc:identifier.uri"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor.advisor","label":"Advisor","values":["Agrawal, Pulkit"]},{"key":"dc:contributor.department","label":"Department","values":["Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science"]},{"key":"dc:creator","label":"Author","values":["Fong, Alisha"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.accessioned","label":"Dc Date Accessioned","values":["2023-07-31T19:30:55Z"]},{"key":"dc:date.available","label":"Dc Date Available","values":["2023-07-31T19:30:55Z"]},{"key":"dc:date.issued","label":"Date","values":["2023-06"]},{"key":"dc:publisher","label":"Institution","values":["Massachusetts Institute of Technology"]},{"key":"dc:type","label":"Dc Type","values":["Thesis"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Master","Master of Engineering in Electrical Engineering and Computer Science"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:rights","label":"Dc Rights","values":["In Copyright - Educational Use Permitted","Copyright retained by author(s)"]},{"key":"dc:rights.uri","label":"Rights URI","values":["https://rightsstatements.org/page/InC-EDU/1.0/"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier.uri","label":"Identifier URI","values":["https://hdl.handle.net/1721.1/151317"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["Recent works have show the promise of LLMs for generalizable task planning. Challenges in integrating LLM for high-level planning include outputting infeasible or sub-optimal plans, but the potentials include cultural commonsense to reason about high-level tasks on par with a human. Related works will generate free-form text that may contain actions inaccessible to the robot or over constrain the planner by providing it a static set of possible actions to select from and output mediocre plans. Humans can also decide when a task is infeasible due to limitations in the action space, and try to propose alternative plans. We show that LMs are also able to. We present an LLM-planner with the ability to request online learning of skills to output and execute optimal tabletop manipulation plans, even when the initial set of robot skills is insufficient. We build a fullstack system and deploy our method in simulation and hardware to demonstrate the capabilities of the planner and the preference for these plans over others in our ablation experiments. To support the learning of new skills, we present a low-level control API conditioned on natural language using Neural Descriptor Fields (NDFs) for out-of-plane category level manipulation that is SE(3)-equivariant and highly data-efficient to enable online learning."]},{"key":"dc:description.degree","label":"Dc Description Degree","values":["M.Eng."]},{"key":"dc:title","label":"Title","values":["NDF-Based API for Human-assisted Language Planning (HaLP)"]}]}],"canonical_facts":{"dc:contributor.advisor":["Agrawal, Pulkit"],"dc:contributor.department":["Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science"],"dc:creator":["Fong, Alisha"],"dc:date.accessioned":["2023-07-31T19:30:55Z"],"dc:date.available":["2023-07-31T19:30:55Z"],"dc:date.issued":["2023-06"],"dc:description.abstract":["Recent works have show the promise of LLMs for generalizable task planning. Challenges in integrating LLM for high-level planning include outputting infeasible or sub-optimal plans, but the potentials include cultural commonsense to reason about high-level tasks on par with a human. Related works will generate free-form text that may contain actions inaccessible to the robot or over constrain the planner by providing it a static set of possible actions to select from and output mediocre plans. Humans can also decide when a task is infeasible due to limitations in the action space, and try to propose alternative plans. We show that LMs are also able to. We present an LLM-planner with the ability to request online learning of skills to output and execute optimal tabletop manipulation plans, even when the initial set of robot skills is insufficient. We build a fullstack system and deploy our method in simulation and hardware to demonstrate the capabilities of the planner and the preference for these plans over others in our ablation experiments. To support the learning of new skills, we present a low-level control API conditioned on natural language using Neural Descriptor Fields (NDFs) for out-of-plane category level manipulation that is SE(3)-equivariant and highly data-efficient to enable online learning."],"dc:description.degree":["M.Eng."],"dc:identifier.uri":["https://hdl.handle.net/1721.1/151317"],"dc:publisher":["Massachusetts Institute of Technology"],"dc:rights":["In Copyright - Educational Use Permitted","Copyright retained by author(s)"],"dc:rights.uri":["https://rightsstatements.org/page/InC-EDU/1.0/"],"dc:title":["NDF-Based API for Human-assisted Language Planning (HaLP)"],"dc:type":["Thesis"],"thesis:degree_name":["Master","Master of Engineering in Electrical Engineering and Computer Science"]},"updated_at":"2026-07-22T22:21:00Z"}