Scalable hierarchical evolution strategies

Hierarchical reinforcement learning (HRL) has been steadily growing in popularity for solving the hardest reinforcement learning problems. However, current HRL algorithms are relatively slow and brittle to hyperparameter changes. This paper offers a solution to these slow and brittle HRL algorithms, by investigating a novel method combining Scalable Evolution Strategies (SES) and HRL. S-ES, named for its excellent scalability, was popularised by Open AI when they showed its performance to be comparable to state-of-the art policy gradient methods. However, S-ES has not been tested in conjunction with HRL methods, which empower temporal abstraction thus allowing agents to tackle more challenging problems. We introduce a novel method merging S-ES and HRL, which creates a highly scalable and fast (wall-clock time) algorithm. We demonstrate that S-ES needs no hyper-parameter tuning for the HRL tasks tested and is indifferent to delayed rewards. This results in a method that is significantly faster than gradient-based HRL methods while having competitive task performance. We extend this method using transfer learning with the aim of increasing task performance and novelty search with the goal of improving its exploration characteristics. The paper's main contribution is thus a novel evolutionary HRL method, namely Scalable Hierarchical Evolution Strategies, which yields greater learning speed and competitive task-performance compared to state-of-the-art gradient-based methods, across a range of tasks.

Grantor

Department of Computer Science

Year dc:date.issued

2022

Author dc:creator

• Abramowitz, Sasha

Advisor dc:contributor.advisor

• Nitschke, Geoff

• Computer Science

Handle dc:identifier.uri

http://hdl.handle.net/11427/36999

OAI identifier oai:identifier

oai:open.uct.ac.za:11427/36999