Multi-agent reinforcement learning: A mean-field perspective

Recent advancements in Reinforcement Learning (RL) have enabled significant progress in solving complex decision-making problems across various domains such as robotics, autonomous driving, and strategy games. However, many real-world scenarios involve multiple agents cooperating and competing with each other to achieve common goals, necessitating the development of Multi-Agent RL (MARL) algorithms. MARL not only addresses the challenges posed by interactions among agents but also enables robust learning in dynamic environments. Mean-Field Games (MFGs) offer a promising framework to tackle scalability issues in RL for multiple decision-making agents by considering the limiting case where the number of agents approaches infinity. Originating from seminal works, MFGs have seen extensive research, extensions, and applications in diverse fields. The thesis focuses on extending RL techniques for purely competitive games (Chapters 2, 3 and 6), Cooperative-Competitive (CC) games (Chapter 4) and the robust N -agent cooperative control problem (Chapter 5). Within purely competitive games Chapter 2 deals with a consensus problem where the agents are split into multiple populations. Although the thesis primarily deals with the Linear Quadratic (LQ) framework (which has applications in finance and engineering), we dedicate Chapter 3 to a general large population Markov game where we relax the assumption of access to a population (mean-field) simulator prevalent in literature. Chapter 4 is regarding a CC game where the agents are divided into multiple teams where these is intra-team cooperation and inter-team competition. Chapter 5 pertains to a purely cooperative control problem where the agents’ dynamics and cost functions can be manipulated by an adversary. This chapter takes the min-max approach by characterizing optimal policies in the presence of the worst adversarial manipulation. Chapter 6 investigates the effects of entropy regularization on cost functions of competitive agents and how it results in exploratory noise in the control policies of the agents. In each of these chapters we start with a literature review highlighting existing approaches and gaps in research. Then we characterize the various equilibria corresponding to each of the problems, followed by RL algorithms to compute these equilibria (in a data driven manner) along with finite sample guarantees and numerical validations.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Mechanical Engineering

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2024

Author dc:creator

• Zaman, Muhammad Aneeq Uz

Contributors dc:contributor

• Başar, Tamer
• Dullerud, Geir
• Jiang, Nan
• Srikant, Rayadurgam

• Multi-agent Reinforcement Learning
• Mean-field Game Theory
• Policy Gradient Methods
• Actor-critic Methods
• Cooperative-competitive Games

Statement dc:rights

• Copyright 2024 Muhammad Aneeq Uz Zaman

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/127124

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F127124