Seeing through Space and Time: Asynchronous Event Processing for Robots

Vision sensors are the eyes of robotic systems, facilitating perception, planning and interaction with the world. Event perception offers a revolutionary new sensing modality to enable real-time perception for autonomous systems. Similar to how humans and animals perceive the world, an event-based sensor measures the continuous brightness changes instead of capturing absolute intensity frames like traditional camera sensors. These sensors are ideal for robotics applications due to several advantages: continuous output with microsecond resolution, High Dynamic Range (HDR) imaging, less motion blur, real-time response without frame rate limitations and low power consumption. Despite their widespread use in autonomy applications, current approaches to processing event data often involve accumulating events over short temporal windows to create image-like pseudo-frames. This compromises temporal resolution and real-time data performance while increasing computational and memory complexity. This thesis aims to provide a general architecture for modelling event streams and their uncertainties using stochastic models, and explore the unique spatial-temporal characteristics of event cameras to develop linear filters for high-speed robotics applications. I present case studies addressing four real-world challenges: (1) Hybrid Event-Frame Fusion for High-Speed HDR Video Reconstruction, (2) Event Data Pre-processing, (3) High-Speed Visual Tracking, (4) High Data Rate Optical Communication. These algorithms process each event independently upon arrival, preserving rich temporal information and improving computational efficiency. Their asynchrony and efficiency make them well-suited for direct implementation on low-level hardware like Field Programmable Gate Arrays (FPGAs) and Application-Specific Integrated Circuits (ASICs) which could further reduce latency and enhance real-time processing capabilities at the hardware level. To take a step toward this vision, this thesis implements asynchronous event data processing on CPUs, addressing the lack of suitable hardware at the project's outset. As event cameras become more capable, affordable, and supported by suitable low-level hardware, I hope that this thesis helps bridge the gaps and drive the field closer to the era of real-time and parallel event data processing on low-level chips integrated into cameras or robotic systems.

Author dc:creator

• Wang, Ziwei

Language dc:language.iso

en_AU

Handle dc:identifier.uri

https://hdl.handle.net/1885/733764963

OAI identifier oai:identifier

oai:openresearch-repository.anu.edu.au:1885/733764963