Improvement in Frame Prediction using Optical Flow

Future frame prediction is a difficult but useful problem to solve in deep learning. The technology can be used to predict future occurrences in a video, anticipate anomalies, and aid autonomous devices in smart decision making. Although there is potential with frame prediction technology, there is still progress that needs to be made with it. As the predicted frame becomes farther away from the last input frame, the image becomes blurry and distorted. This indicates that the model is more uncertain about the motion occurring in the image frame. To reduce model uncertainty shown in predictions, optical flow information from each video was extracted and combined with the video frames. An optical flow-based approach is uncommon in frame prediction and has not been implemented with a fully Convolutional Neural Network (CNN) based architecture. In this work, the change in image quality evaluation metrics and overall image quality is analyzed across 4 different datasets between a state-of-the-art frame prediction model and a modified model that combines optical flow information. The results demonstrate that adding optical flow information improves the model Mean Squared Error (MSE) by 4.11% and its Structural Similarity Index Metric (SSIM) by 0.41% for the Moving MNIST dataset. Optical flow improved the SSIM value of Taxi BJ, KTH, and KITTI by 0.02%, 0.011%, and 1.297% respectively. While there was a consistent improvement in performance, the models still need more improvement in terms of the quality of images predicted in the distant future.

Name thesis:degree_name

Master of Science

Level thesis:degree_level

masters

Discipline thesis:degree_discipline

Computer Engineering

Department dc:contributor.department

Electrical and Computer Engineering

Grantor dc:publisher

Virginia Tech

Year dc:date.issued

2023

Author dc:creator

• Wormack, Craig Frederick Luther Jr.

Chair dc:contributor.committeechair

• Jones, Creed F. III

Committee members dc:contributor.committeemember

• Abbott, Amos L.
• Wyatt, Chris L.

• Optical flow
• Frame prediction

Statement dc:rights

• In Copyright

Licence dc:rights.uri

• http://rightsstatements.org/vocab/InC/1.0/

Language dc:language.iso

en

Dc Identifier Other

vt_gsexam:37784

OAI identifier oai:identifier

oai:vtechworks.lib.vt.edu:10919/115671