TranDynaMo: A comparative study of transformer and GRU performance in modeling dynamical systems

Dynamics modeling is critical to many robotics and control tasks and applications, such as motion planning and trajectory optimization. However, obtaining a model through current methods is difficult and expensive, so instead, there have been many efforts to come up with data-driven methods to learn dynamics models. With the popularity and success of Large Language Models (LLMs) such as ChatGPT, we were inspired to find a way to leverage transformers, the building blocks of ChatGPT, to solve the dynamics learning problem. In this thesis, we study the viability of utilizing transformers for dynamics modeling. We do this by first introducing transformers and posing dynamics learning as a sequence-to-sequence modeling task. We then create a GPT2-based dynamics learning framework called TranDynaMo and test its dynamics modeling performance for various dynamical systems, such as chaotic systems and systems with limit cycles, by giving it an initial condition and seeing how well it can predict the trajectory. We then compare TranDynaMo against a GRU-based framework. We find that when the dynamics learning framework is required to predict parts of a trajectory significantly beyond the horizon of the training trajectories it was given, TranDynaMo beats the GRU-based framework; however, if the training and testing trajectories are of almost the same horizon, both perform satisfactorily, but the GRU-based framework outperforms TranDynaMo, thus indicating that the generalization of transformers is better.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Electrical & Computer Engr

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2023

Author dc:creator

• Pandey, Amogh

Contributors dc:contributor

• Mehr, Negar

• Transformers, Large Language Models
• Chatgpt
• Gpt
• Llm
• Dynamics Learning
• Dynamics Modeling
• Rnn
• Gru
• Control System Dynamics
• Dynamical Models

Statement dc:rights

• Copyright 2023 Amogh Pandey

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/122269

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F122269