Training and Inference in Early-Exit Deep Q-Networks for Efficient Reinforcement Learning

Neural networks have become instrumental in various machine learning tasks, but their increasing complexity poses challenges in terms of computational resources and real-time decision-making. To address these challenges, this thesis explores the integration of early-exit neural networks (EENNs) with reinforcement learning (RL), a novel approach that has received limited attention in the literature. Early-exit strategies introduce intermediate exit points in neural networks, allowing for dynamic decision-making based on the model’s level of confidence. Reinforcement learning, on the other hand, enables autonomous systems to make intelligent decisions through sequential decision-making. We hypothesize that combining early exits with reinforcement learning can enhance learning efficiency and generalization. In this study, we design and implement Early Exit Deep Q-Network (EEDQN), a framework that integrates EENNs with RL algorithms. Through a series of experiments using benchmark datasets and complex environments, we assess the approach on two OpenAI Gym tasks with distinct input regimes: CartPole (vector state) and Atari Pong (pixels). Across both domains, EEDQN yields results that preserve baseline return while lowering expected per-decision FLOPs, with larger savings when many states are “easy” and confidently handled by the early exit. These results demonstrate that conditional computation via early exits can make RL policies adaptive to state difficulty without sacrificing control, minimizing overthinking, delays, and power utilization.

Author dc:creator

• Michael Mishal (23291719)

• Engineering
• Electronics and Electrical

Statement dc:rights

• In Copyright

DOI dc:identifier

https://doi.org/10.25417/uic.31451482.v1

OAI identifier oai:identifier

oai:figshare.com:article/31451482