Hierarchical Game Theory based Control for Large Scale Multi-Agent Systems: A Hybrid Reinforcement Learning Approach

Distributed optimal control and coordination strategies for large-scale multi-agent systems (LS-MAS) operating under uncertainty have become increasingly critical due to the challenges of scalability, communication complexity, and computational intractability. Traditional control approaches face the curse of dimensionality as the number of interacting agents increases, making them unsuitable for real-time implementation in dense environments. To address these limitations, mean field game (MFG) theory has been employed as a scalable approximation tool, wherein the behavior of the overall population is represented through a probability density function (PDF). However, conventional MFG formulations often lack the ability to capture diverse coordination behaviors due to their reliance on a single global PDF, leading to suboptimal performance in heterogeneous or structured systems. To enhance flexibility and coordination efficiency, a hierarchical game-theoretic framework is introduced by decomposing the entire agent population into multiple interacting subgroups. Within this structure, inter-group and intra-group dynamics are governed through layered game formulations that incorporate both strategic leadership and population-level interactions. The framework enables scalable solutions to complex coordination problems such as flocking and formation, under uncertainty and in a distributed manner. To further improve coordination effectiveness, an extended formulation of MFG is developed to account for structural heterogeneity and asymmetric density constraints. A decomposition strategy is proposed to represent the global terminal behavior through a collection of subgroup-level distributions, allowing increased expressiveness and adaptability in control design. In addition, a reinforcement learning-based algorithm is designed to achieve distributed policy optimization within this hierarchical structure, enabling agents to adapt to changing environments with limited information exchange. Finally, a theoretical and empirical study is conducted to analyze the fundamental trade-off between coordination efficiency and computational complexity. Stability, convergence, and robustness of the proposed approach are validated through extensive simulations and comparative analysis.

Level thesis:degree_level

Doctorate Degree

Year dc:date.issued

2025

Author dc:creator

• Dey, Shawon

Advisor dc:contributor.advisor

• Xu, Hao

Committee members dc:contributor.committeemember

• Fadali, M. Sami
• Commuri, Sesh
• Fu, Xingang
• Voulgaris, Petros

Language dc:language.iso

en_US, English

Repository record dc:identifier.uri

https://scholarwolf.unr.edu/handle/11714/11567

OAI identifier oai:identifier

oai:scholarwolf.unr.edu:11714/11567