Attention-Enabled Reinforcement Learning for Control of Scalable Multi-Agent Systems

Multi-agent reinforcement learning has been the subject of considerable interest and effort for its potential as a means of specifying behavior policies for multi-agent systems. Specifically, on-policy algorithms based on gradient estimation have achieved state-of-the-art performance on end-to-end control problems once thought beyond the scope of machine learning methods. In seeking to apply the benefits of MARL to practical control of physical autonomous systems, we must begin to account for three factors: (1) the presence of other autonomous elements in the environment configuration space, which may or may not be amenable to coordination; (2) non-idealities in sensing the configuration of the environment (e.g. locality and limited observability); and (3) variability in the number of sensed dynamical elements. The attention head, a relational ML structure originally designed for extraction of abstract natural language features, is structurally well suited to addressing these challenges. This work presents a systematic argument and framework for the use of attention as an input layer to enable learning of neural policy models in changing multi-agent environments which are not well-suited to other representations. In benchmark physical simulations, it is shown that such models achieve competitive performance on cooperative and mixed cooperative/competitive MAS control tasks as the agent cohort is arbitrarily changed. Prospective advantages of attention-based architectures for physical autonomous systems in select applications are discussed, as well as drawbacks associated with explainability and potential for emergent behavior.

Name thesis:degree_name

Master of Science

Level thesis:degree_level

Master's Degree

Grantor

University of Nevada, Reno

Year dc:date.issued

2024

Author dc:creator

• Dailey, Joseph A.

Advisor dc:contributor.advisor

• Xu, Hao

Committee members dc:contributor.committeemember

• Fadali, M. Sami
• La, Hung

• Attention
• Autonomous Systems
• Intelligent Control
• Multi-agent Systems
• Reinforcement Learning

Statement dc:rights

• Creative Commons Attribution-ShareAlike 4.0 International

Repository record dc:identifier.uri

https://scholarwolf.unr.edu/handle/11714/10949

OAI identifier oai:identifier

oai:scholarwolf.unr.edu:11714/10949