Research map / Label-efficient and robust learning / Recognition and long-tail

Data Distributional Properties As Inductive Bias for Systematic Generalization

CVPR 2025 · Label-efficient and robust learning › Recognition and long-tail · approach: Data augmentation and generalization bias · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that training-data distributional properties (color diversity, burstiness, latent intervention) induce systematic generalization in a small multimodal Transformer on a CLEVR-like masked-language task, linked to lower NMI and greater representational parallelism.

Problem

Deep networks fail at systematic generalization to unseen combinations of latent attributes, and the role of training data distributional properties as inductive biases for SG in multimodal models is understudied.

Approach

Treats data distribution properties (diversity of attribute values, burstiness limiting values per image, and random latent intervention) as inductive biases, and explains their effect via NMI between attributes and the parallelism (p-score) of learned representations.

Key results

• Increasing color diversity from 8 to 216 colors yields up to an 89% absolute OOD accuracy gain on the shape task.
• Burstiness raises shape OOD accuracy by up to 14.8% while reducing color task accuracy by up to 14.3%.
• Latent intervention (hue jitter) increases shape OOD accuracy by up to 15%; p-score correlates with OOD shape accuracy at Pearson 0.73.

Measured effects reported in the paper

Surprising findings

• Adding more in-distribution training data (with 8 colors) slightly degrades OOD performance rather than improving it.
• Lack of model capacity does not explain the systematic generalization gains from diversity.

Limitations

• Experiments are on a synthetic CLEVR-like dataset with only three shapes and limited attributes.
• Burstiness improves shape generalization at the cost of color task performance.

Experimental setup

Datasets: CLEVR-like synthetic dataset generated with CLEVR code

Benchmarks: CLEVR-like systematic split (in-distribution and out-of-distribution)

Metrics: accuracy p-score NMI DCI disentanglement DCI completeness

Baselines: 8-color baseline model

Builds on: CLEVR Chan et al. burstiness in-context learning Abbe et al. NMI and systematic generalization in CLIP p-score parallelism metric

Closest papers by meaning

• Position: Deep Learning is Not So Mysterious or Different (ICML 2025 · similarity 0.88)
• Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Data (NeurIPS 2024 · similarity 0.87)
• Feature Contamination: Neural Networks Learn Uncorrelated Features and Fail to Generalize (ICML 2024 · similarity 0.86)
• Changing the Training Data Distribution to Reduce Simplicity Bias Improves In-distribution Generalization (NeurIPS 2024 · similarity 0.86)
• What Variables Affect Out-of-Distribution Generalization in Pretrained Models? (NeurIPS 2024 · similarity 0.86)
• When does compositional structure yield compositional generalization? A kernel theory. (ICLR 2025 · similarity 0.86)