Neural Network Fusion of Multi-Modal Sensor Data For Autonomous Surface Vessels

<p>Autonomous surface vessels (ASV) can potentially improve the safety of vessels traditionally operated by humans. Despite advancements in autonomous on-road vehicles, many of these advancements have yet to be realized for ASVs. This is primarily due to lacking ASV sensing platforms and public datasets for ASV-based perception research. To that end, this dissertation demonstrates the design of a synchronized multi-modal sensing platform for ASVs utilizing GPS/INS, LiDAR, LWIR cameras, HDR camera, and high-resolution cameras. The sensing platform is designed to maximize the overlap of sensors for multi-modal research and provides accurate intrinsic and extrinsic calibration between each sensor. Furthermore, the GPS provides precise time stamp capabilities for each sensor. This sensing platform is used for data collection to create a novel multi-modal marine dataset with LiDAR ground truth annotations. Finally, the dataset is used to demonstrate the feasibility of deep learning for semantic segmentation in a marine environment. This is done with three separate models: a LiDAR-only model, a camera-only model, and a deep fusion model. The LiDAR model demonstrates the feasibility of using LiDAR image projections as a valuable data representation. The camera model demonstrates the feasibility of cross-modality learning with a camera input and LiDAR ground truth. Finally, the fusion network demonstrates the advantages of combining features between multiple modalities to improve model performance. While each model showcases its strengths, future work may explore improved model optimization, integrating transformers and using temporal features to detect moving objects better.</p>

Name thesis:degree_name

Doctor of Philosophy in Mechanical Engineering

Level thesis:degree_level

Dissertation - Open Access

Discipline thesis:degree_discipline

Mechanical Engineering

Year

2023

Author dc:creator

• Thompson, David J

• Autonomous surface vessels
• safety
• GPS
• LiDAR
• LWIR
• HDR camera
• Other Mechanical Engineering
• Robotics

Repository record dc:identifier

https://commons.erau.edu/edt/741

OAI identifier oai:identifier

oai:commons.erau.edu:edt-1761