Research map / Vision-language and multimodal / Multimodal LLMs
CoIDO: Efficient Data Selection for Visual Instruction Tuning via Coupled Importance-Diversity Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoIDO, a lightweight MLP scorer trained on 20% of LLaVA-665K that jointly optimizes sample importance and cluster-level diversity via homoscedastic uncertainty weighting, selecting a 20% subset that reaches 98.2% of full-data LoRA fine-tuning performance on LLaVA-1.5-7B.
Problem
Visual instruction tuning on large datasets like LLaVA-665K is computationally expensive, and existing data selection methods require processing the full dataset with the target MLLM and treat importance and diversity separately.
Approach
A plug-in scorer co-trained with the target MLLM on a 20% random subset assigns each sample a weight that is trained with an importance loss (softmax-weighted cross-entropy) and a diversity loss (variance of cluster-mean weights from spectral clustering), balanced by learned homoscedastic uncertainty terms; the trained scorer then ranks the full dataset without extra selection algorithms.
Key results
- CoIDO with 20% of LLaVA-665K reaches Rel. 98.2% of full-data LoRA fine-tuning across ten benchmarks, matching ICONS (97.1% in authors' reproduction) and above COINCIDE (97.4%) and TIVE (96.7%).
- CoIDO reports the lowest total training FLOPs (4.2E) among compared methods, versus 10.2E for full data and 12.6E for ICONS.
- On Vision-Flan, CoIDO achieves Rel. 102.1% over full fine-tuning, and a scorer trained on LLaVA-665K transfers to reach 103.7%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select with CoIDO instead of COINCIDE | LLaVA-1.5-7B-LoRA on LLaVA-665K · LLaVA-665K, 20% selection | Rel. (%): 97.4 -> 98.2 | Helped |
| Select with CoIDO instead of random sampling | LLaVA-1.5-7B-LoRA on LLaVA-665K · LLaVA-665K, 20% selection | Rel. (%): 95.1 -> 98.2 | Helped |
| Use CoIDO instead of random selection | LLaVA-1.5-7B-LoRA on Vision-Flan · Vision-Flan, 20% selection | Rel. (%): 97.8 -> 102.1 | Helped |
| Transfer scorer trained on LLaVA-665K without retraining | CoIDO on Vision-Flan · Vision-Flan, 20% selection | Rel. (%): 102.1 -> 103.7 | Helped |
| Add diversity loss during training rather than optimizing importance alone (ablation) | CoIDO optimization on LLaVA-665K · LLaVA-665K, 20% selection | Rel. (%): 89.0 -> 98.2 | Helped |
| Use homoscedastic uncertainty optimization instead of simple loss summation (ablation) | CoIDO optimization on LLaVA-665K · LLaVA-665K, 20% selection | Rel. (%): 92.0 -> 98.2 | Helped |
| Use homoscedastic uncertainty optimization instead of learnable weighted losses (ablation) | CoIDO optimization on LLaVA-665K · LLaVA-665K, 20% selection | Rel. (%): 95.9 -> 98.2 | Helped |
| Include both CLIP features and the three quality/alignment scores (ablation) | CoIDO scorer on LLaVA-665K · LLaVA-665K, 20% selection | Rel. (%): 96.3 -> 98.2; 96.4 -> 98.2 | Helped |
| Set cluster count to 20 rather than 5 (ablation) | CoIDO on LLaVA-665K · LLaVA-665K, 20% selection | Rel. (%): 92.2 -> 98.2 | Helped |
| Set cluster count to 20 rather than 10 (ablation) | CoIDO on LLaVA-665K · LLaVA-665K, 20% selection | Rel. (%): 94.2 -> 98.2 | Helped |
| Set cluster count to 20 rather than 40 (ablation) | CoIDO on LLaVA-665K · LLaVA-665K, 20% selection | Rel. (%): 97.4 -> 98.2 | Helped |
| Use MLP scorer instead of attention-only scorer (ablation) | CoIDO on LLaVA-665K · LLaVA-665K, 20% selection | Rel. (%): 96.1 -> 98.2 | Helped |
Surprising findings
- A scorer trained on LLaVA-665K and applied to Vision-Flan outperformed a scorer trained directly on Vision-Flan.
- The selected 20% subset slightly outperforms full-data fine-tuning on Vision-Flan (102.1%), suggesting full-data training is not always best.
Limitations
- Relative gains over the strongest baselines are small (98.2% vs 97.4% COINCIDE and 97.1% ICONS in the authors' reproduction), and some benchmarks show losses versus full data.
- Evaluation of LLaVA-Bench switched from gpt-3.5-turbo to gpt-4o-mini, and the ICONS baseline reproduction differed from the original report (97.1% vs 98.6%).
- Relies on several hand-picked hyperparameters (cluster count M=20, 20% training ratio) and a DeepSeek-V3 LLM score for text quality, adding external model dependencies.
Experimental setup
Datasets: LLaVA-665K Vision-Flan LLaVA-150K
Benchmarks: VQAv2 GQA VizWiz SQA-I TextVQA POPE MME MMBench MMBench-cn LLaVA-Bench
Metrics: Rel. (%) relative performance accuracy total FLOPs MLLM training data cost
Baselines: Random CLIP-Score EL2N Perplexity SemDeDup D2-Pruning Self-Sup Self-Filter TIVE ICONS COINCIDE
Builds on: Kendall et al. multi-task uncertainty weighting LLaVA LoRA CLIP
Closest papers by meaning
- Adapt-$\infty$: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection (ICLR 2025 · similarity 0.90)
- Data Selection for Fine-tuning Vision Language Models via Cross Modal Alignment Trajectories (ICML 2026 · similarity 0.89)
- What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning (ICLR 2024 · similarity 0.88)
- LESS: Selecting Influential Data for Targeted Instruction Tuning (ICML 2024 · similarity 0.87)
- LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning (CVPR 2026 · similarity 0.87)
- DELIFT: Data Efficient Language model Instruction Fine-Tuning (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.