Research map / Detection and segmentation / Open-set and few-shot detection
Vision CNNs trained to estimate spatial latents learned similar ventral-stream-aligned representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains ResNet CNNs on synthetic TDW images to estimate spatial latents (distance, translation, rotation) versus object categories, finding spatial-latent models reach comparable ventral-stream neural alignment and similar early/middle-layer representations.
Problem
Whether the primate ventral stream is optimized only for object categorization, or whether training on spatial latents such as object position, distance, and pose yields comparably brain-aligned vision models and similar representations.
Approach
Uses a 3D graphics engine (ThreeDWorld) to generate synthetic images with ground-truth spatial and category latents, holds architecture and data fixed, and compares Brain-Score neural alignment and CKA representational similarity across models trained on different latent subsets; proposes non-target latent variability as an explanation for convergence.
Key results
- Distance-only ResNet-50 (one output unit) reaches neural mean 0.399, about 97% of the all-classification-plus-all-spatial model (0.411) and close to ImageNet-1K (0.430).
- Object category classification on TDW-117 reaches 0.412 neural mean, versus 0.430 for ImageNet-1K-trained ResNet-50.
- Models trained on different spatial and category latents show highly similar CKA representations in early and middle layers, diverging mainly in late layers; rotation diverges most.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on synthetic object categories rather than ImageNet-1K | ResNet-50 · Brain-Score public benchmarks | neural mean: 0.430 -> 0.412 | Hurt |
| Train ResNet-50 to estimate object distance rather than classify ImageNet-1K | ResNet-50 · Brain-Score public benchmarks | neural mean: 0.430 -> 0.399 | Hurt |
| Add translation estimation to distance estimation (ablation) | ResNet-50 · Brain-Score public benchmarks | neural mean: 0.399 -> 0.407 | No clear effect |
| Add rotation estimation to distance estimation (ablation) | ResNet-50 · Brain-Score public benchmarks | neural mean: 0.399 -> 0.402 | No clear effect |
| Add rotation estimation to translation estimation (ablation) | ResNet-50 · Brain-Score public benchmarks | neural mean: 0.405 -> 0.403 | No clear effect |
| Train jointly on distance, translation, and rotation (ablation) | ResNet-50 · Brain-Score public benchmarks | neural mean: 0.399 -> 0.405 | No clear effect |
| Train on spatial distance instead of object category classification | ResNet-50 · Brain-Score public benchmarks | neural mean: 0.412 vs 0.399 | No clear effect |
| Add category variability to training data while predicting spatial targets (ablation) | spatial latent-trained CNNs · TDW synthetic images | non-target category decoding performance | Helped |
| Add translation variability to training data while predicting other targets (ablation) | latent-trained CNNs · TDW synthetic images | non-target translation decoding performance | Mixed |
| Increase translation variability while predicting object category (ablation) | category-trained CNNs · TDW synthetic images | translation decoding performance | Mixed |
Surprising findings
- Models trained to estimate only one spatial latent (distance) achieve neural alignment nearly matching models trained on hundreds of categories, contrary to a scaling hypothesis.
- Spatial-latent-trained models have much lower behavioral alignment than category-trained models, which the authors attribute partly to the behavioral benchmark favoring categorization.
Limitations
- Synthetic-trained models still fall short of ImageNet-trained neural alignment, and the synthetic dataset lacks natural image nuances.
- Spatial-latent models underperform category models on some non-public IT benchmards and behavioral alignment, and evaluation relies mainly on regression-based neural metrics.
- The non-target latent variability mechanism is only partially supported; other factors such as rendering quality and image statistics may contribute.
Experimental setup
Datasets: TDW-117 TDW-N (N=2,4,6,8,16) ImageNet-1K
Benchmarks: FreemanZiemba2013public.V1-pls FreemanZiemba2013public.V2-pls MajajHong2015public.V4-pls MajajHong2015public.IT-pls Rajalingham2018public-i2n
Metrics: Brain-Score neural alignment behavioral alignment CKA linear decoding accuracy Pearson correlation
Baselines: Object category classification Object identity classification ImageNet-1K classification Untrained models
Builds on: Brain-Score ThreeDWorld centered kernel alignment ResNet
Closest papers by meaning
- Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream (ICML 2025 · similarity 0.90)
- Sparse components distinguish visual pathways & their alignment to neural networks (ICLR 2025 · similarity 0.90)
- Bridging Critical Gaps in Convergent Learning: How Representational Alignment Evolves Across Layers, Training, and Distribution Shifts (NeurIPS 2025 · similarity 0.88)
- STSBench: A Large-Scale Dataset for Modeling Neuronal Activity in the Dorsal Stream of Primate Visual Cortex (NeurIPS 2025 · similarity 0.87)
- L-WISE: Boosting Human Visual Category Learning Through Model-Based Image Selection and Enhancement (ICLR 2025 · similarity 0.87)
- A Convolutional Neural Network Interpretable Framework for Human Ventral Visual Pathway Representation (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.