The Science of Mind Reading: New Inverse Optimal Control Framework

Continuous control and planning by the brain remain poorly understood and is a major challenge in the field of Neuroscience. To truly say that we understand the underlying mechanisms we should first be able to explain the behavioral actions of the animals, so that we can relate the neural activity to these explanations. We hypothesize that animals choose actions rationally under possibly mistaken assumptions about the world. That is, their actions result from solving an optimal control problem. We consider a naturalistic task to study this in greater detail, under a formal optimal control framework of Partially Observable Markov Decision Processes. In our "firefly" task, monkeys are trained to steer to catch transiently visible fireflies in a Virtual Reality environment, using motion cues to navigate. There are no spatial landmarks in this task, which introduces significant uncertainty. The animal must therefore make decisions to maximize its total reward based on beliefs about the hidden firefly location. We cannot observe this internal belief state, nor the internal model assumed by the animal, but only the actions chosen and the sensory observations the animal received. To explain the actions we need to reconstruct the internal model which results in the actions. Using reinforcement learning algorithms, we solve the forward problem of solving for the optimal actions given a model and a given reward function. We then propose a novel framework of inverse reinforcement learning, which learns optimal policies generalized over the model space. Our proposed method is able to recover the true model of simulated agents within theoretical error bounds. Finally, we interpret our framework in a way that opens new possibilities for hierarchical inference while an animal learns.

Name thesis:degree_name

Master of Science

Level thesis:degree_level

Masters

Discipline thesis:degree_discipline

Engineering

Grantor

Rice University

Year dc:date.issued

2018

Author dc:creator

• Daptardar, Saurabh

Advisor dc:contributor.advisor

• Pitkow, Xaq

• Inverse Reinforcement Learning
• Inverse Optimal Control
• Reinforcement Learning
• Optimal Control
• Neuroscience

Statement dc:rights

• Copyright is held by the author, unless otherwise indicated. Permission to reuse, publish, or reproduce the work beyond the bounds of fair use or other exemptions to copyright law must be obtained from the copyright holder.

Language dc:language.iso

eng

Handle dc:identifier.uri

https://hdl.handle.net/1911/105893

OAI identifier oai:identifier

oai:repository.rice.edu:1911/105893