eMARLIN: Addressing Coordination and Partial Observability in Distributed Reinforcement Learning for Traffic Signal Control

Traffic congestion is a growing concern in urban areas, pertaining to its significant economic, environmental, and safety impacts. While traditional solutions like infrastructure expansion and public transit development are vital, they involve high budget and time costs. This thesis aims to address congestion by enhancing urban traffic network's efficiency through Intelligent Transportation Systems (ITS), with a specific focus on multi-intersection Adaptive Traffic Signal Control (ATSC). ATSC adapts traffic signal timing based on real-time traffic data, but the increasing granularity and decentralization of control introduce new challenges. These include coordination among decentralized controllers, in-field limitations in distributive communication, and partial observability due to limited detection. This thesis addresses these challenges through a combination of analytical studies, software engineering, and methodology development. First, it identifies common ground in ATSC research by unifying Receding-Horizon Optimization (RHO) and Reinforcement Learning (RL), leading to a clearer understanding of how these methodologies complement each other. Additionally, a multi-scenario, multi-agent traffic control software platform, which is the WOLF project, provides a foundation for comparing various ATSC approaches. Second, to enhance decentralized collaboration, the thesis proposes a novel architecture, eMARLIN, which uses latent space embedding to improve agent coordination while conserving communication bandwidth. This approach addresses the coordination challenge by enabling agents to share compressed information with their neighbours, promoting effective collaboration. Third, the thesis addresses partial observability, which is considered as a systemic issue in ATSC. It identifies two sources of lack-of-sensing: limited detection capabilities and constrained detection range. System modelling and domain knowledge relaxation provide a comprehensive understanding of the problem's structure characteristics. Building upon the eMARLIN architecture, techniques like incorporating historical data sequences and Long Short-Term Memory (LSTM) networks are employed to mitigate these issues. Additionally, the integration of Transformer-based models into eMARLIN demonstrates further advancements in handling temporal information and enhancing coordination among agents. Overall, this thesis contributes to the development of efficient, scalable, and effective decentralized ATSC solutions, condensed into the family of eMARLIN algorithms. The proposed methodologies and software platform offer practical and empirical support for advancing ATSC research, with implications for improving traffic management in urban areas. This work provides a roadmap for future research, laying the groundwork for continued exploration in intelligent transportation systems.

Department dc:contributor.department

Civil Engineering

Year dc:date.issued

2024

Author dc:creator

• Wang, Xiaoyu

Advisors dc:contributor.advisor

• Abdulhai, Baher
• Sanner, Scott

• Adaptive traffic signal control
• Control systems
• Markov decision processes
• Multi-agent reinforcement learning
• Partial observability

Statement dc:rights

• Attribution-NonCommercial 4.0 International

Licence dc:rights.uri

• http://creativecommons.org/licenses/by-nc/4.0/

Handle dc:identifier.uri

http://hdl.handle.net/1807/140619

OAI identifier oai:identifier

oai:utoronto.scholaris.ca:1807/140619