Introducing builder perspective in minecraft architect utterance generation

The task of Collaborative Dialogue in Minecraft involves developing interactive agents that can communicate with humans to collaboratively solve tasks in grounded scenarios. Previous attempts to solve this task have treated the tasks of Architect Utterance and Builder Action Prediction as separate problems, and independent models have been developed to solve each. However, a human architect would typically think about how to build the target structure before giving verbal instructions to the Builder. Therefore, in this work, we focus on the Architect Utterance generation task and explore how the Builder Action Prediction model can be used to improve utterance generation. We achieve this by using a customized builder that can view the Minecraft world through the Architect’s perspective and generate action sequences. These action sequences represent a simulation of what the Architect would do if they were acting as the Builder, allowing the Architect to think in terms of Builder actions. We leverage this Builder-For-Architect model to introduce action sequences and their effects on the world state into the Architect’s decision-making process.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2023

Author dc:creator

• Athani, Amit A

Contributors dc:contributor

• Hockenmaier, Julia

• Nlp
• Natural Language Processing
• Cnn
• Roi

Statement dc:rights

• Copyright 2023 Amit Athani

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/120121

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F120121