Geometry-based video prediction with camera motion for mobile robotics

Video prediction is one of the fundamental research problems in computer vision, and it has a wide range of applications in planning and control for robotics. Recent learning-based approaches show promising results on various video datasets, and some have seen successful applications in planning robot arm motion. However, predicting future observations in a sequence given a set of past images remains a challenging task in mobile robotics, especially when the camera is in motion. Early works in this area use deterministic approaches, which often yield visually unintuitive results due to the intrinsic variability of motion in the future. More recent works have adopted stochastic models to generate sharper future frames. However, most methods do not account for camera motion and perform poorly in scenarios with moving cameras when they are deployed on vehicles and mobile robots. To solve the challenging task of video prediction on mobile platforms, we propose a geometry-based prediction framework that combines visual odometry prediction and view synthesis. Based on a sequence of observed frames, our method first predicts future camera poses and extracts the 3D geometry of the world, which are then jointly used to generate predicted future frames. Specifically, we train a recurrent visual odometry prediction model conditioned on raw RGB images with ground-truth pose labels. In addition, we train SynSin, a view synthesis method to generate 2D images from novel viewpoints using a 3D world representation. By combining these approaches, we demonstrate that our hierarchical deterministic approach outperforms previous stochastic works on the KITTI dataset.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Electrical & Computer Engr

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2022

Author dc:creator

• Liang, Weihang

Contributors dc:contributor

• Driggs-Campbell, Katherine Rose

• Video Prediction
• Computer Vision
• Robotics
• Machine Learning

Statement dc:rights

• Copyright 2022 Weihang Liang

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/115799

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F115799