Research map / Vision-language and multimodal / Human and embodied modeling
Does Data Scaling Lead to Visual Compositional Generalization?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Controlled (n,k) experiments train ResNet-50s from scratch and probe pretrained DINO/CLIP to show visual compositional generalization depends on combinatorial data diversity rather than data scale, and that linearly factored representations enable generalization from two combinations per concept value.
Problem
Whether scaling data and model size yields compositional generalization to unseen attribute-object combinations in vision models, given sparse concept co-occurrence in large web datasets.
Approach
Vary the number of concept values n and training combinations per value k in a controlled grid to separate data diversity from data quantity, and link zero-shot compositional generalization to linearly factored (additive) representations, proving that such structure needs only two combinations per concept value for perfect generalization.
Key results
- Increasing ID training data 4x (7.5k to 30k or up to 120k) leaves OOD accuracy drops of 60-80% on unseen combinations.
- Linear structure (R^2 > 0.8, cosine similarity < 0.1-0.2) and above-90% zero-shot accuracy emerge only at high combination diversity (75-100% of combinations).
- Pretrained CLIP and DINOv2 reach >90% accuracy on some concepts under linear factorization with k=2, but no model achieves perfect generalization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the number of target classes while fixing diagonal training combinations | ResNet-50 trained from scratch · Unseen concept combinations | zero-shot accuracy | Helped |
| Increase the number of training combinations while fixing the maximum target classes | ResNet-50 trained from scratch · Unseen concept combinations | zero-shot accuracy | Helped |
| Increase in-distribution training data quantity within the same distribution, up to 4x | ResNet-50 trained from scratch · Unseen concept combinations | accuracy: 4x; 60-80% | No clear effect |
| Train with high combination diversity | ResNet-50 trained from scratch · Unseen concept combinations | zero-shot accuracy: above 90% | Helped |
| Increase combination diversity from limited to moderate | ResNet-50 trained from scratch · Unseen concept combinations | zero-shot accuracy: 60-80% | Helped |
| Increase combination diversity from limited to moderate | ResNet-50 trained from scratch · Balanced data covering all concept combinations | decoded accuracy: <80% -> 100% | Helped |
| Increase combination diversity to high levels | ResNet-50 trained from scratch · Feature representation reconstruction | R^2: R^2>0.8 | Helped |
| Increase combination diversity to high levels | ResNet-50 trained from scratch · Concept representations | cosine similarity: cosine similarity <0.1 | Helped |
| Use pretraining rather than training from scratch | Pretrained vision models · Unseen concept combinations | average accuracy | Helped |
| Increase the diversity of downstream training combinations | Pretrained vision models · Unseen concept combinations | average accuracy | Helped |
| Use linear factorization to recover concept representations from two combinations per concept value | Ideally linearly factored representations · All unseen concept combinations | accuracy: k=2 | Helped |
| Use pretrained features with recovered linear concept representations | CLIP or DINOv2 · Color, orientation, digit, and background concepts | accuracy: > 90% | Mixed |
Surprising findings
- Increasing in-distribution training data quantity does not improve OOD compositional generalization, even with perfect ID accuracy.
- Pretrained models outperform from-scratch ResNet50 but still struggle, and pretraining is not a substitute for data diversity.
Limitations
- Theory assumes linearly independent factors and balanced training data; may break down as number of concept values grows.
- Experiments restricted to two concepts and single-object images.
- Oracle model selection on the test set favors models.
Experimental setup
Datasets: DSprites 3DShapes PUG Colored-MNIST FSprites
Benchmarks: DSprites 3DShapes PUG Colored-MNIST FSprites
Metrics: zero-shot accuracy on unseen combinations R^2 linearity orthogonality (cosine similarity) decodability accuracy
Baselines: ResNet50-ImageNet1K ResNet50-DINOv1 DINOv2-ViT-L/14 CLIP-ViT-L/14 ResNet-50 from scratch
Builds on: linearly factored embeddings oracle model selection
Closest papers by meaning
- Scalable Evaluation and Neural Models for Compositional Generalization (NeurIPS 2025 · similarity 0.92)
- When does compositional structure yield compositional generalization? A kernel theory. (ICLR 2025 · similarity 0.92)
- Scaling can lead to compositional generalization (NeurIPS 2025 · similarity 0.92)
- When and How Does CLIP Enable Domain and Compositional Generalization? (ICML 2025 · similarity 0.90)
- Provable Compositional Generalization for Object-Centric Learning (ICLR 2024 · similarity 0.90)
- Compositional Generalization via Forced Rendering of Disentangled Latents (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.