Research map / Vision-language and multimodal / Multimodal LLMs
COMPACT: COMPositional Atomic-to-Complex Visual Capability Tuning
Xindi Wu, Hee Seung Hwang, Polina Kirichenko, Esin Tureci, Olga Russakovsky
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes COMPACT, a visual instruction tuning data curation approach that selects samples by task complexity (compositions of atomic visual capabilities) to efficiently finetune multimodal LLMs.
Problem
Visual instruction tuning datasets are randomly sampled image-question pairs without regard to how informative each pair is; selecting informative subsets for efficient MLLM finetuning remains open, especially regarding task complexity.
Approach
Explores the impact of task complexity on informative data curation and introduces compositional atomic-to-complex capability tuning, curating VIT data by combining atomic visual capabilities into more complex samples; only the abstract was available, so details beyond that are not stated.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train on COMPACT (32K compositional + 5% VIT) instead of random 65K subset | LLaVA-v1.5-7B · 8-benchmark average | relative performance: 95.38 -> 100.18 | Improved |
| train on COMPACT instead of full LLaVA-665K | LLaVA-v1.5-7B · MM-Vet | score: 29.22 -> 31.74 | Improved |
| train on COMPACT instead of full LLaVA-665K | LLaVA-v1.5-7B · MME | score: 1478.48 -> 1379.94 | Hurt |
| increase sample complexity by raising k from 2 to 3 (k-matching control) (ablation) | COMPACT 49K · 8-benchmark average | relative performance: 97.55 -> 98.83 | Improved |
| use compositional data matched to LLaVA k distribution instead of random subset (ablation) | LLaVA-v1.5-7B 49K · 8-benchmark average | relative performance: 96.28 -> 97.55 | Improved |
| sample kgen=1,2,3 instead of only kgen=3 (ablation) | COMPACT 16K · MM-Vet, LLaVA-W, MMStar | score | Improved |
| add a small VIT subset (about 1-5%) to compositional data (ablation) | COMPACT 16K · MM-Vet, SeedBench2Plus, LLaVA-W | score | Improved |
| raise VIT subset from 5% to 7% (ablation) | COMPACT 16K · multimodal benchmarks | relative performance | No clear effect |
| use open-source Qwen3-VL-4B generator instead of Gemini (ablation) | COMPACT 65K · 8-benchmark average | relative performance: 100.18 -> 98.31 | Hurt |
| remove scene understanding from capability pool (ablation) | COMPACT 16K · avg relative performance | relative drop: -5.2% | Hurt |
| remove shape capability from pool (ablation) | COMPACT 16K · avg relative performance | relative drop: -0.7% | No clear effect |
| train on higher-k data (ablation) | COMPACT · MMStar k=3 questions | relative gain: +22.7% | Improved |
Closest papers by meaning
- Adapt-$\infty$: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection (ICLR 2025 · similarity 0.91)
- VIGC: Visual Instruction Generation and Correction (AAAI 2024 · similarity 0.90)
- CoIDO: Efficient Data Selection for Visual Instruction Tuning via Coupled Importance-Diversity Optimization (NeurIPS 2025 · similarity 0.90)
- TaskGalaxy: Scaling Multi-modal Instruction Fine-tuning with Tens of Thousands Vision Task Types (ICLR 2025 · similarity 0.90)
- Learning to Instruct for Visual Instruction Tuning (NeurIPS 2025 · similarity 0.89)
- AVA-Bench: Atomic Visual Ability Benchmark for Vision Foundation Models (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
