Probing the Mechanisms of Reinforcement Learning: Reinforcement Learning, Ventral Striatal Astrocytes, and the Dynamic Coordination of Information Seeking with Learning

While reinforcement learning has been a vital component in artificial intelligence and machine learning, there exist many open questions about its implementations and how to improve them, in both minds and machines. Among these are i) the contribution of non-neuronal cell types to reinforcement learning, and ii) information-seeking behavior during reinforcement learning. In this thesis, we studied these main topics pertaining to reinforcement learning. In the first chapter, we examined the role of astrocytes in reinforcement learning, and in the second, we investigated human information seeking during reinforcement learning. Neurons in the human and animal brain have been known to support value-based decision-making and reinforcement learning in the brain [222, 131, 7]. However, to date, the role of astrocytes in reinforcement learning has not been thoroughly studied. Here, we trained mice on a bandit task, and attenuated the astrocyte activity in Ventral Striatum (VS), Dorsolateral Striatum (DLS), and Dorsomedial Striatum (DMS). We found that mice, whose astrocytes in VS were targeted, showed decreased bandit performance and increased win-stay behavior. Throughout computational modeling, we showed that these patterns could be explained by increased decision randomness. We then showed that these observations could be recapitulated by the deep neural network simulations, by attenuating input sharing across units. In the 2���� chapter, we investigate human information seeking during reinforcement learning. Information-seeking (IS) refers to intrinsic motivation towards obtaining information and is a critical aspect of human and animal behavior. While IS has been studied in the context of decision making, to the best of our knowledge, a thorough study of IS in the context of RL did not exist to date. We found that humans pay monetary rewards for obtaining information during RL. Moreover, their IS increases with reward uncertainty, decreases with learning, and correlates negatively with bandit performance. Furthermore, extensive reinforcement learning modeling revealed that IS correlates with the degree of random exploration, and having access to early information during learning increases the speed of learning in both humans and artificial neural networks. Overall, our work demonstrates a specific algorithmic relation between non-neuronal cell activity and reinforcement learning, and the coordination between human information seeking during reinforcement learning for the first time. xii

Name thesis:degree_name

Doctor of Philosophy (PhD)

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Electrical & Systems Engineering

Year dc:date.available

2025

Author dc:creator

• Sogukpinar, Fatih

Contributors dc:contributor

• Ilya Monosov
• Gaia Tavoni; Michael Frank; Naoki Hiratani; ShiNung Ching

• Astrocytes;Exploration;Information seeking;Reinforcement Learning
• Engineering
• Systems Science

Language dc:language

English (en)

Identifier

https://openscholarship.wustl.edu/eng_etds/1319

OAI identifier oai:identifier

oai:openscholarship.wustl.edu:eng_etds-2396