Research map / Architectures and efficiency / RNNs and normalization
Scalable Evaluation and Neural Models for Compositional Generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an efficient orthotopic evaluation framework for compositional generalization (constant training runs in the number of attributes), benchmarks >5000 supervised vision models, and introduces Attribute Invariant Networks that approach explicitly disentangled models at much lower parameter overhead.
Problem
Compositional generalization in supervised vision models lacks standardized, rigorous evaluation protocols, and existing evaluations are either shallow or combinatorially expensive; general-purpose vision architectures lack the inductive biases for it while bias-injecting approaches scale poorly.
Approach
Orthotopic evaluation generates compositional splits by iteratively projecting and pruning the data in every c-dimensional attribute subspace, so a single training run covers all attributes, with a compositional similarity index c controlling difficulty. Attribute Invariant Networks use attribute-specific encoders and heads around a shared meta-model so each encoder's gradients are invariant to ot
Key results
- AINs improve average compositional generalization accuracy over monolithic ResNet-18 by +23.43% while adding only 6.4%-16% parameter overhead, versus 300%-600% for ED models.
- Orthotopic evaluation yields test accuracy on average 11.6% lower than pair-wise evaluation, with the same overall trends across model families.
- On the c=0 extrapolation split no model generalized, while nearly all models reached perfect in-distribution accuracy at c=I.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add attribute-invariant gradient updates using AIN | ResNet-18 · Shapes3D compositional generalization split | test accuracy (%): 85.47$^{\pm 0.71}$ -> 85.26$^{\pm 0.63}$ | Hurt |
| add attribute-invariant gradient updates using AIN | ResNet-18 · MPI3D compositional generalization split | test accuracy (%): 41.59$^{\pm 2.16}$ -> 54.02$^{\pm 1.04}$ | Helped |
| add attribute-invariant gradient updates using AIN | ResNet-18 · dSprites compositional generalization split | test accuracy (%): 21.43$^{\pm 0.63}$ -> 61.43$^{\pm 0.33}$ | Helped |
| add attribute-invariant gradient updates using AIN | ResNet-18 · I-RAVEN compositional generalization split | test accuracy (%): 11.30$^{\pm 2.80}$ -> 66.23$^{\pm 2.26}$ | Helped |
| add attribute-invariant gradient updates using AIN | ResNet-18 · Cars3D compositional generalization split | test accuracy (%): 33.90$^{\pm 1.84}$ -> 43.90$^{\pm 0.79}$ | Helped |
| add attribute-invariant gradient updates using AIN | ResNet-18 · CLEVR compositional generalization split | test accuracy (%): 24.68$^{\pm1.86}$ -> 54.30$^{\pm2.15}$ | Helped |
| replace original activation functions with learnable activations (ablation) | vision models in orthotopic evaluation · compositional generalization splits | test accuracy | No clear effect |
| increase optimization steps up to 240× (ablation) | vision models in orthotopic evaluation · compositional generalization splits | test accuracy | No clear effect |
| use orthotopic evaluation instead of pair-wise evaluation | vision models · compositional generalization evaluation | test accuracy: 11.6\% on average lower | Hurt |
Surprising findings
- Pre-trained models generally perform worse than models trained from scratch on compositional splits.
- Learnable activation functions and longer training (grokking regime) did not improve compositional generalization.
Limitations
- Requires access to ground-truth generative factor labels, limiting applicability to real-world noisy datasets.
- Neither AIN attribute-invariant gradients nor ED guarantee perfect disentanglement.
- Experiments limited to ResNet-18 backbone for AIN comparisons and to small synthetic and semi-synthetic datasets.
Experimental setup
Datasets: dSprites I-RAVEN Shapes3D CLEVR Cars3D MPI3D
Benchmarks: dSprites I-RAVEN Shapes3D CLEVR Cars3D MPI3D
Metrics: exact-match accuracy
Baselines: MLP ResNet DenseNet Wide ResNet ConvNeXt ViT Swin Transformer explicitly disentangled (ED) models
Builds on: Shared Architecture pair-wise compositional evaluation Recombination-to-Range
Closest papers by meaning
- Does Data Scaling Lead to Visual Compositional Generalization? (ICML 2025 · similarity 0.92)
- When does compositional structure yield compositional generalization? A kernel theory. (ICLR 2025 · similarity 0.90)
- Scaling can lead to compositional generalization (NeurIPS 2025 · similarity 0.89)
- Provable Compositional Generalization for Object-Centric Learning (ICLR 2024 · similarity 0.88)
- When and How Does CLIP Enable Domain and Compositional Generalization? (ICML 2025 · similarity 0.87)
- Compositional Generalization via Forced Rendering of Disentangled Latents (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.