Autonomous UAV positioning using multi-agent reinforcement learning with decentralized swarms

The rapid development of unmanned aerial vehicles (UAVs) has aroused public attentions, as the versatility of UAVs allows it to be easily adapted for various activities. In recent years, we start seeing more and more UAVs being applied for research and commercial use. In the meantime, the development of Reinforcement Learning has facilitated more intelligent drone behaviors, such as self-flying drone control and autonomous drone racing. With the recent development in multi-agent reinforcement learning and wireless communi- cations, it is more interesting to explore what UAV swarms can do more than a single UAV itself. In this work, we first apply deep reinforcement learning algorithms in multi-agent domains for decentralized UAV swarms, in which each UAV acts as an independent agent having full control of its own actions. We model the environment using Graph Neural Network and attention-based embedding. This proposed method forms a fixed-size encoding for environments with different number of variables including drones and landmarks, which allows it to scale to different number of UAVs, and makes it feasible for efficient transfer learning. We then corroborate the effectiveness of this method in various different tasks through experiments. UAV interactions in a swarm can be categorized into cooperation and competition, and thus environments are ground as fully-cooperative, fully-competitive, and mixed environments. We conduct experiments on two tasks for drone swarms, with the first one being a fully-cooperative environment, and the second a mixed cooperative-competitive environment. The first environment is called DroneConnect, where UAVs are being used as relays to connect mobile devices in remote areas to set up a temporary communication networks among them. We design and implement a drone-relay system that allows drone swarm to cooperatively maximize coverage for these mobile devices. We compare the coverage ratio of centralized reinforcement learning, decentralized multi-agent reinforcement learning method to an alternative optimization method. We then introduce a team of adversaries into the second use case, namly DroneCombat, where two drone swarms defense and oppose each other autonomously. We investigate the performance of the graph-based method and observe the natural emergence of complex behaviors. We also simulate a more real-world scenario where agents are partial-observable of their local neighborhood instead of being omniscient of the entire environment. With extensive experiments, we show that this vision and sensing limitation can be mitigated by message passing through communication.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2023

Author dc:creator

• Chen, Yifan

Contributors dc:contributor

• Caesar, Matthew

• Uav
• Reinforcement Learning

Statement dc:rights

• Copyright 2023 Yifan Chen

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/120474

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F120474