AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

COMPACT: COMPositional Atomic-to-Complex Visual Capability Tuning

Xindi Wu, Hee Seung Hwang, Polina Kirichenko, Esin Tureci, Olga Russakovsky

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Visual instruction tuning

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes COMPACT, a visual instruction tuning data curation approach that selects samples by task complexity (compositions of atomic visual capabilities) to efficiently finetune multimodal LLMs.

Problem

Visual instruction tuning datasets are randomly sampled image-question pairs without regard to how informative each pair is; selecting informative subsets for efficient MLLM finetuning remains open, especially regarding task complexity.

Approach

Explores the impact of task complexity on informative data curation and introduces compositional atomic-to-complex capability tuning, curating VIT data by combining atomic visual capabilities into more complex samples; only the abstract was available, so details beyond that are not stated.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
train on COMPACT (32K compositional + 5% VIT) instead of random 65K subsetLLaVA-v1.5-7B · 8-benchmark averagerelative performance: 95.38 -> 100.18Improved
train on COMPACT instead of full LLaVA-665KLLaVA-v1.5-7B · MM-Vetscore: 29.22 -> 31.74Improved
train on COMPACT instead of full LLaVA-665KLLaVA-v1.5-7B · MMEscore: 1478.48 -> 1379.94Hurt
increase sample complexity by raising k from 2 to 3 (k-matching control) (ablation)COMPACT 49K · 8-benchmark averagerelative performance: 97.55 -> 98.83Improved
use compositional data matched to LLaVA k distribution instead of random subset (ablation)LLaVA-v1.5-7B 49K · 8-benchmark averagerelative performance: 96.28 -> 97.55Improved
sample kgen=1,2,3 instead of only kgen=3 (ablation)COMPACT 16K · MM-Vet, LLaVA-W, MMStarscoreImproved
add a small VIT subset (about 1-5%) to compositional data (ablation)COMPACT 16K · MM-Vet, SeedBench2Plus, LLaVA-WscoreImproved
raise VIT subset from 5% to 7% (ablation)COMPACT 16K · multimodal benchmarksrelative performanceNo clear effect
use open-source Qwen3-VL-4B generator instead of Gemini (ablation)COMPACT 65K · 8-benchmark averagerelative performance: 100.18 -> 98.31Hurt
remove scene understanding from capability pool (ablation)COMPACT 16K · avg relative performancerelative drop: -5.2%Hurt
remove shape capability from pool (ablation)COMPACT 16K · avg relative performancerelative drop: -0.7%No clear effect
train on higher-k data (ablation)COMPACT · MMStar k=3 questionsrelative gain: +22.7%Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy