Feynman-Kac Numerical Techniques for Stochastic Optimal Control

Three significant advancements are proposed for improving numerical methods in the solution of forward-backward stochastic differential equations (FBSDEs) appearing in the Feynman-Kac representation of the value function in stochastic optimal control (SOC) problems. First, we propose a novel characterization of FBSDE estimators as either on-policy or off-policy, highlighting the intuition for these techniques that the distribution over which value functions are approximated should, to some extent, match the distribution the policies generate. Second, two novel numerical estimators are proposed for improving the accuracy of single-timestep updates. In the case of LQR problems, we demonstrate both in theory and in numerical simulation that our estimators result in near machine-precision level accuracy, in contrast to previously proposed methods that can potentially diverge on the same problems. Third, we propose a new method for accelerating the global convergence of FBSDE methods. By the repeated use of the Girsanov change of probability measures, it is demonstrated how a McKean-Markov branched sampling method can be utilized for the forward integration pass, as long as the controlled drift terms are appropriately compensated in the backward integration pass. Subsequently, a numerical approximation of the value function is proposed by solving a series of function approximation problems backwards in time along the edges of a space-filling tree.

Level thesis:degree_level

Doctoral

Department dc:contributor.department

Interactive Computing

Grantor dc:publisher

Georgia Institute of Technology

Year dc:date.issued

2021

Author dc:creator

• Hawkins, Kelsey Pal

Advisor dc:contributor.advisor

• Tsiotras, Panagiotis

Committee members dc:contributor.committeemember

• Berenson, Dmitry
• Coogan, Samuel
• Theodorou, Evangelos
• Vamvoudakis, Kyriakos

• Stochastic optimal control
• Forward-backward stochastic differential equations
• Robotics
• Partial differential equations
• Motion planning
• Rapidly exploring random trees

Language dc:language.iso

en_US

Handle dc:identifier.uri

http://hdl.handle.net/1853/66058

OAI identifier oai:identifier

oai:repository.gatech.edu:1853/66058