Representational alignment of humans and machines for computer vision
type: CreativeWork
description: Recent progress in computer vision has led to deep neural networks that match or even surpass human-level performance across various machine learning tasks. However, we identify a significant difference between the representations learned by these models and human conceptual understanding. We introduce a novel approximate Bayesian method for learning object concept representations from human behavior in a triplet odd-one-out task. The method uses variational inference to produce sparse, non-negative representations with uncertainty estimates, improving the reproducibility of object dimensions and the consistency in predicting human behavior. These mental representations can downstream be used to unravel the differences between human and neural network representations of object concepts. Our findings reveal that contemporary computer vision models, while achieving human-level performance, fail to align with human mental representations. Factors such as training data and objective function appear to play a crucial role in alignment, while the model architecture and number of parameters have minimal impact. We pinpoint the reason for this misalignment: human conceptual knowledge is hierarchically organized, while traditional training methods embed similar images close together, thereby often ignoring the global organization of object concepts. To address this mismatch, we developed a novel method that improves the global structure of these representations by linearly aligning them with human similarity judgments, leading to improved performance in few-shot learning and anomaly detection tasks. However, linearly aligning model representation can only improve the representations of large image/text models that have seen billions of images during pretraining but it fails to achieve the same for smaller models trained on less data. By training a teacher model to imitate human judgments and subsequently distilling this human-like structure into pretrained student models, we…
abstract: Recent progress in computer vision has led to deep neural networks that match or even surpass human-level performance across various machine learning tasks. However, we identify a significant difference between the representations learned by these models and human conceptual understanding. We introduce a novel approximate Bayesian method for learning object concept representations from human behavior in a triplet odd-one-out task. The method uses variational inference to produce sparse, non-negative representations with uncertainty estimates, improving the reproducibility of object dimensions and the consistency in predicting human behavior. These mental representations can downstream be used to unravel the differences between human and neural network representations of object concepts. Our findings reveal that contemporary computer vision models, while achieving human-level performance, fail to align with human mental representations. Factors such as training data and objective function appear to play a crucial role in alignment, while the model architecture and number of parameters have minimal impact. We pinpoint the reason for this misalignment: human conceptual knowledge is hierarchically organized, while traditional training methods embed similar images close together, thereby often ignoring the global organization of object concepts. To address this mismatch, we developed a novel method that improves the global structure of these representations by linearly aligning them with human similarity judgments, leading to improved performance in few-shot learning and anomaly detection tasks. However, linearly aligning model representation can only improve the representations of large image/text models that have seen billions of images during pretraining but it fails to achieve the same for smaller models trained on less data. By training a teacher model to imitate human judgments and subsequently distilling this human-like structure into pretrained student models, we…
author: Muttenthaler, Lukas
language: en
license: https://creativecommons.org/licenses/by/4.0/
date published: 2025
date modified: 2026-07-27T21:28:40Z
provider: Technische Universität Berlin
publisher: Networked Digital Library of Theses and Dissertations
type: WebPage
source organization: Technische Universität Berlin

PropertyValue
property ID: DOI
value: 10.14279/depositonce-23490