Communication and generalization in multi-agent learning

Multi-agent learning aims to allow artificial intelligence (AI) agents to learn from interactions with other agents in an environment. However, as AI increasingly integrates into real-world systems, significant challenges arise in how to robustly interact with and communicate with a variety of other agents, particularly in complex environments such as autonomous driving, where humans and AI agents coexist. This dissertation research investigates how agents can be trained to effectively communicate with and generalize to diverse partners (including humans) in simulated real-world scenarios. Towards addressing this challenge, this dissertation explores three key dimensions: (1) learning communication-supporting representations that facilitate coordination, (2) developing multi-agent policies that generalize to new teammates or opponents, and (3) learning to collaborate with human-like agents or to use human language. This dissertation makes novel contributions along each dimension. First, the dissertation presents Coopernaut, a framework that learns compact, transmittable representations from local observations to support communication among autonomous vehicles under bandwidth constraints. It also introduces LLM+Debrief, which enables embodied agents to coordinate in driving scenarios by generating and interpreting natural language messages, paving the way for human-compatible agent communication. Second, it introduces MACTA, a reinforcement learning and game-theoretic training framework that produces robust policies capable of generalizing to unseen and adaptive opponents. In addition, L-BRDiv is introduced as a teammate generation strategy that promotes behavioral diversity during training, improving generalization and performance in ad hoc teamwork settings. Third, the dissertation investigates mixed-autonomy traffic coordination through decentralized training in environments with both human-proxy and AI agents. Empirical results demonstrate that even a small number of trained autonomous vehicles can collaborate effectively to influence human behavior and improve overall traffic efficiency without requiring centralized control. Collectively, these contributions advance multi-agent AI by unifying communication, generalization, and human-AI collaboration. Evaluated in both toy domains and realistic simulated environments, primarily focusing on autonomous driving and hardware security, the work demonstrates how agents can adapt to novel partners and communicate effectively in human-interpretable ways.

Name thesis:degree_name

Doctor of Philosophy

Discipline thesis:degree_discipline

Electrical and Computer Engineering

Grantor

The University of Texas at Austin

Year dc:date.issued

2025

Author dc:creator

• Cui, Jiaxun

Advisor dc:contributor.advisor

• Stone, Peter, 1971-

Committee members dc:contributor.committeemember

• Chinchali, Sandeep
• Zhu, Yuke
• Zhang, Amy
• Tian, Yuandong

• Generalization
• Communication
• Multi-agent learning
• Reinforcement learning

Language dc:language.iso

English

Identifier URI

https://doi.org/10.26153/tsw/61968

OAI identifier oai:identifier

oai:repositories.lib.utexas.edu:2152/134646