Research map / 3D vision / 3D generation
Evaluating Multiview Object Consistency in Humans and Image Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MOCHI, a human-vs-vision-model benchmark based on odd-one-out and match-to-sample 3D shape inference triplets, finding humans far outperform DINOv2, CLIP, and MAE models while model and human performance remain correlated.
Problem
Whether computer vision models represent the 3D structure of objects, evaluated by aligning their zero-shot viewpoint-invariant shape judgments with human observers rather than only average accuracy.
Approach
Adapts a cognitive-science multiview object matching paradigm (odd-one-out and match-to-sample triplets across four stimulus sets) into a benchmark that compares model and human accuracy, reaction time, and gaze, using a lightweight same-different SVM readout on frozen features.
Key results
- Humans achieve 78% average accuracy versus 44% for the best model (DINOv2-Giant), with humans outperforming all models by a wide margin.
- DINOv2 models average 0.43 normalized accuracy, CLIP 0.29, and MAE -0.03; MAE performance hovers around chance regardless of scale.
- Human-model performance is correlated across trials (r=.35), and humans spend more reaction time on trials where DINOv2-G fails (r=-0.29).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DINOv2-G instead of human observers | MOCHI object consistency task · MOCHI | normalized accuracy: 0.78 -> 0.44 | Hurt |
| Use DINOv2 rather than CLIP or MAE | Vision models evaluated on MOCHI · MOCHI | normalized accuracy | Helped |
| Increase model size from DINOv2-Base to DINOv2-Giant | DINOv2 · MOCHI | normalized accuracy: 32% to 44% | Helped |
| Increase model size | CLIP · MOCHI | normalized accuracy | Helped |
| Increase model size | MAE · MOCHI | normalized accuracy | No clear effect |
| Use the lightweight same-different classifier instead of distance metrics | Vision model readout · MOCHI | normalized accuracy: t(2018) = 12.83, P = 2.74x10^-36 | Helped |
| Use the lightweight same-different classifier instead of a standard linear probe | Vision model readout · MOCHI | normalized accuracy | Helped |
| Use a standard linear probe rather than a lightweight same-different classifier | Vision model readout · MOCHI | normalized accuracy | Hurt |
Surprising findings
- MAE models perform at or below chance at all sizes, and scaling does not improve them.
- DINOv2 model self-attention does not predict human gaze patterns better than chance.
Limitations
- Model performance gap to humans is large, and the lightweight same-different probe choice was selected post hoc as best-performing readout.
- Gaze data collected only on the barense subset of stimuli.
- Model-human correlation is weak at trial level (noise ceiling fractions below 0.4%).
Experimental setup
Datasets: barense stimuli majaj stimuli shapenet stimuli shapegen stimuli
Benchmarks: MOCHI
Metrics: accuracy (normalized to chance and ceiling) split-half reliability reaction time correlation gaze salience correlation
Baselines: CLIP MAE LLaVA
Builds on: odd-one-out task (cognitive science) match-to-sample task
Closest papers by meaning
- When does perceptual alignment benefit vision representations? (NeurIPS 2024 · similarity 0.91)
- Contour Integration Underlies Human-Like Vision (ICML 2025 · similarity 0.89)
- The 3D-PC: a benchmark for visual perspective taking in humans and machines (ICLR 2025 · similarity 0.88)
- Not Too Generative, Not Too Discriminative: The Human Alignment Sweet Spot (NeurIPS 2026 · similarity 0.88)
- Model–Behavior Alignment under Flexible Evaluation: When the Best-Fitting Model Isn’t the Right One (NeurIPS 2025 · similarity 0.87)
- Cross-modal Associations in Vision and Language Models: Revisiting the Bouba-Kiki Effect (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.