Subdominance Minimization: A Satisficing Perspective on Imitation Learning

Human decision-making often hinges on a delicately-balanced interplay between multiple, potentially conflicting objectives. However, prevailing imitation learning methods tend to prioritize optimizing a single imitation objective. This myopic focus on a singular objective frequently leads to unintended and undesirable behaviors in learned models. For example, an autonomous vehicle prioritizing travel time over adherence to traffic laws, or indeed a language model learning to generate increasingly creative responses at the expense of factual accuracy. We instead adopt the recently-proposed notion of subdominance which, given an arbitrary number of cost objectives to minimize simultaneously, quantifies the cost of choosing one solution relative to a reference solution. We present a novel focused satisficing approach to imitation learning, using subdominance to seek imitator policies that the demonstrator may consider acceptable, rather than optimal. We leverage both offline and online subdominance minimization to focus policy learning on parts of the demonstrator's trajectories which are hardest to imitate. In the domain of offline reinforcement learning, we present another approach for training decision transformers offline via subdominance minimization which allows us to learn autoregressive policies even in the absence of a ground truth reward function. Finally, we present an approach for language-conditioned, multitask imitation learning, where we leverage learned instruction representations to train language-conditioned subdominance minimizing policies.

Author dc:creator

• Rushit Shah (24400088)

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning
• Imitation Learning
• Decision Theory
• Behavioral Economics

Statement dc:rights

• In Copyright
• Open Access after 2028-05-01

DOI dc:identifier

https://doi.org/10.25417/uic.32995130.v1

OAI identifier oai:identifier

oai:figshare.com:article/32995130