Research map / Vision-language and multimodal / Human and embodied modeling
Iterated Learning Improves Compositionality in Large Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IL-CLIP, an iterated-learning training scheme that periodically respawns the language agent in a codebook-based CLIP to improve vision-language compositionality.
Problem
Vision-language models like CLIP perform poorly on compositional understanding (e.g., distinguishing attribute-object bindings and word order), and scaling does not fix this.
Approach
Reframes CLIP contrastive learning as a Lewis Signaling Game and applies cultural transmission by periodically resetting the language agent (with distillation from a frozen shared codebook and a limited interaction phase), so representations become easier to learn and more compositional.
Key results
- IL-CLIP trained on CC3M improves standard CLIP's mean compositionality score from 28.3 to 34.4 (about +4.7% on SugarCrepe per the abstract).
- On CC12M, IL-CLIP reaches mean compositionality 38.0 vs 34.9 for CLIP, matching NegCLIP (38.0) without training on compositional negatives.
- IL-CLIP keeps zero-shot classification roughly on par with CLIP (CC3M mean 24.2 vs 21.7; CC12M 38.5 vs 35.7 across 18 datasets).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use iterated learning with a shared codebook rather than standard CLIP | CLIP trained on CC3M · Compositional benchmarks (mean) | retrieval R@1: 28.3 -> 34.4 | Helped |
| use iterated learning with a shared codebook rather than standard CLIP | CLIP trained on CC12M · Compositional benchmarks (mean) | retrieval R@1: 34.9 -> 38.0 | Helped |
| use iterated learning on top of Codebook-CLIP | Codebook-CLIP trained on CC3M · Image classification datasets (mean) | top-1 accuracy: 24.4 -> 24.2 | Hurt |
| use iterated learning on top of Codebook-CLIP | Codebook-CLIP trained on CC12M · Image classification datasets (mean) | top-1 accuracy: 38.8 -> 38.5 | Hurt |
| use 3k rather than 6k steps per generation (ablation) | IL-CLIP on CC3M · Compositional benchmarks | COMP: 34.4 -> 32.1 | Hurt |
| use 12k rather than 6k steps per generation (ablation) | IL-CLIP on CC3M · Compositional benchmarks | COMP: 34.4 -> 32.9 | Hurt |
| reset the vision agent instead of the language agent (ablation) | IL-CLIP on CC3M · Compositional benchmarks | COMP: 34.4 -> 33.9 | Hurt |
| reset language and vision agents alternatively (ablation) | IL-CLIP on CC3M · Compositional benchmarks | COMP: 34.4 -> 30.7 | Hurt |
| do not freeze codebook weights during distillation (ablation) | IL-CLIP on CC3M · Compositional benchmarks | COMP: 34.4 -> 31.9 | Hurt |
| remove the shared codebook (ablation) | IL-CLIP on CC3M · Compositional benchmarks | COMP: 34.4 -> 28.0 | Hurt |
| replace iterated learning with Lipschitz regularization (ablation) | CLIP on CC3M · Compositional benchmarks | COMP: 34.4 -> 27.8 | Hurt |
Surprising findings
- Resetting both agents between generations hurts performance, and resetting the language agent works better than resetting the vision agent.
- NegCLIP, despite explicit compositional hard negatives, performs notably worse than standard CLIP on zero-shot classification.
Limitations
- Training is unstable due to randomness in spawning new agents.
- Evaluated only at CLIP-scale (ViT-B/32, CC3M/CC12M), not large-scale models.
- Compositionality is assessed indirectly; interpretability claims for codebook are partly qualitative.
Experimental setup
Datasets: CC3M CC12M
Benchmarks: SugarCrepe CREPE Cola Winoground ImageNet1k CIFAR-100 CIFAR-10 STL-10 VOC2007 Caltech101 SUN397 Pets
Metrics: R@1 retrieval accuracy zero-shot top-1 accuracy in-batch matching accuracy estimated Lipschitz upper bound
Baselines: CLIP Codebook-CLIP NegCLIP Lipschitz-regularized CLIP
Builds on: CLIP Codebook-CLIP NegCLIP Lewis Signaling Game Sparsemax
Closest papers by meaning
- Unveiling the Compositional Ability Gap in Vision-Language Reasoning Model (NeurIPS 2025 · similarity 0.90)
- Diagnosing the Compositional Knowledge of Vision Language Models from a Game-Theoretic View (ICML 2024 · similarity 0.88)
- Not Only Text: Exploring Compositionality of Visual Representations in Vision-Language Models (CVPR 2025 · similarity 0.88)
- VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models (CVPR 2026 · similarity 0.88)
- Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.88)
- NePTune: A Neuro-Pythonic Framework for Tunable Compositional Reasoning on Vision-Language (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.