Research map / Vision-language and multimodal / Human and embodied modeling
Evaluating alignment between humans and neural network representations in image-based learning tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Tests 86 pretrained neural networks' representations as features for cognitive models predicting human choices in two few-shot naturalistic image learning tasks (category and reward learning), finding CLIP-style contrastive multimodal models align best.
Problem
Whether and which pretrained neural network representations generalise like humans in few-shot learning of functional/category relationships over naturalistic images, going beyond similarity judgement alignment benchmarks.
Approach
Use pretrained network embeddings as inputs to per-trial linear learners (logistic/Bayesian regression) and score alignment by how well their predictions explain human trial-by-trial choices via McFadden's R^2 in new naturalistic category and reward learning paradigms built on THINGS images.
Key results
- CLIP models were the top 7 (category) and top 6 (reward) of 86 representations in predicting human choices; 14 of 16 better-than-ground-truth representations in category learning were CLIP.
- Larger models (rho=0.48) and models trained on more images (rho=0.52) predicted human choices better.
- Alignment methods (Harmonization, DreamSim, gLocal) gave no consistent improvement in human alignment over matched baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CLIP loss rather than SimCLR loss on YFCC15M | Matched-size vision models · Category and reward learning tasks | McFadden's R² for predicting human choices | Helped |
| Combine CLIP and SimCLR losses | Matched-size vision models · Category and reward learning tasks | McFadden's R² for predicting human choices | Helped |
| Increase model size | Pretrained vision models · Category and reward learning tasks | Human-choice alignment: ρ = 0.48 | Helped |
| Train on more images | Pretrained vision models · Category and reward learning tasks | Human-choice alignment: ρ = 0.52 | Helped |
| Increase class separation in representations | Pretrained vision models · Category and reward learning tasks | Human-choice alignment: ρ = 0.29 | Helped |
| Increase similarity between representations and task features | Pretrained vision models · Category and reward learning tasks | Human-choice alignment: ρ = 0.67 | Helped |
| Reduce intrinsic dimensionality | Pretrained models, most strongly multimodal models · Category and reward learning tasks | Human-choice alignment: ρ = -0.32 | Helped |
| Use CLIP loss instead of SimCLR loss | YFCC15M models · Category and reward learning tasks | Task accuracy | Helped |
| Apply Harmonization | Models with matched-architecture, matched-data baselines · Category and reward learning tasks | Human-choice alignment | Mixed |
| Apply DreamSim alignment | Pretrained models · Category and reward learning tasks | Human-choice alignment | Hurt |
| Apply gLocal alignment | Models with matched-architecture, matched-data baselines · Category and reward learning tasks | Human-choice alignment | Mixed |
| Apply Harmonization | Models with matched-architecture, matched-data baselines · Category and reward learning tasks | Task accuracy | Helped |
Surprising findings
- Lower intrinsic dimensionality correlated with better alignment, driven mainly by multimodal models, contrary to some prior alignment work.
- Larger model size predicted human choice, contrary to previous findings that parameter count is not a robust predictor of human alignment.
Limitations
- Model factors (loss, data, size, architecture) are confounded across the tested pretrained networks, so effects cannot be isolated.
- Only two experimental paradigms with linear task rules based on THINGS embedding dimensions were tested.
- Human alignment measured only with behaviour, not brain data.
Experimental setup
Datasets: THINGS YFCC15M ImageNet THINGS odd-one-out judgements ClickMe NIGHTS Peterson et al. similarity judgements
Benchmarks: THINGS odd-one-out judgements ClickMe Peterson et al. similarity judgements NIGHTS
Metrics: McFadden's R^2 Spearman correlation task accuracy class separation R^2 intrinsic dimensionality (TwoNN) CKA
Baselines: THINGS odd-one-out embedding (generative task features) Harmonization models DreamSim gLocal SimCLR
Builds on: THINGS database CLIP Harmonization gLocal DreamSim
Closest papers by meaning
- Not Too Generative, Not Too Discriminative: The Human Alignment Sweet Spot (NeurIPS 2026 · similarity 0.89)
- Bridging Critical Gaps in Convergent Learning: How Representational Alignment Evolves Across Layers, Training, and Distribution Shifts (NeurIPS 2025 · similarity 0.88)
- Geometry of naturalistic object representations in recurrent neural network models of working memory (NeurIPS 2024 · similarity 0.88)
- When does perceptual alignment benefit vision representations? (NeurIPS 2024 · similarity 0.88)
- Model–Behavior Alignment under Flexible Evaluation: When the Best-Fitting Model Isn’t the Right One (NeurIPS 2025 · similarity 0.87)
- Alignment of CNN and Human Judgments of Geometric and Topological Concepts (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.