Research map / Vision-language and multimodal / Multimodal LLMs
Filter Images First, Generate Instructions Later: Pre-Instruction Data Selection for Visual Instruction Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PreSel, which selects unlabeled images for visual instruction tuning before generating instructions, using a reference-set loss-ratio task-importance estimate and DINOv2 clustering with neighbor centrality; it matches full-data LLaVA-1.5 tuning with 15% of images labeled.
Problem
Visual instruction tuning data selection assumes all images already have instructions, but generating instructions (via GPT API or human labor) for large unlabeled image pools is costly.
Approach
Perform data selection on unlabeled images before instruction generation: estimate per-task importance from a small instructed reference set using an Instruction Relevance Score (ratio of response losses with and without the question), then allocate task-wise budgets and pick representative images via DINOv2 k-means clusters and a neighbor-centrality score.
Key results
- On LLaVA-1.5 with 15% of data, PreSel reaches 97.9% average relative performance vs. full fine-tuning, outperforming Random (95.7%) and the best baseline COINCIDE (95.5%) while generating instructions for only 15% of images.
- On Vision-Flan (191 tasks), PreSel achieves 100.1% relative performance with 15% of data, slightly above full fine-tuning.
- Transfers to LLaVA-Vicuna-13B and Llama-3-8B, outperforming Random by 1.7% and 1.2% relative performance respectively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PreSel instead of full-data fine-tuning | LLaVA-1.5-7B · LLaVA-1.5 | Rel. (%): 100 -> 97.9 | Hurt |
| use PreSel instead of COINCIDE | LLaVA-1.5-7B · LLaVA-1.5 | Rel. (%): 95.5 -> 97.9 | Helped |
| use PreSel instead of full-data fine-tuning | LLaVA-1.5-7B · Vision-Flan | Rel. (%): 100 -> 100.1 | Helped |
| use task-importance rather than size-balanced task allocation (ablation) | PreSel with NC · LLaVA-1.5 | Rel. (%): 97.1 -> 97.9; 0.8% | Helped |
| use task-importance rather than size-balanced task allocation (ablation) | PreSel with random intra-cluster selection · LLaVA-1.5 | Rel. (%): 94.9 -> 96.9; 2.0% | Helped |
| add NC to random within-task selection (ablation) | PreSel with uniform task balancing · LLaVA-1.5 | Rel. (%): 94.8 -> 95.8 | Helped |
| add NC to random within-task selection (ablation) | PreSel with size-balanced task allocation · LLaVA-1.5 | Rel. (%): 94.9 -> 97.1 | Helped |
| add k-means clustering to selection without NC (ablation) | PreSel · LLaVA-1.5 | Rel. (%): 96.2 -> 96.9 | Helped |
| add k-means clustering to NC-based selection (ablation) | PreSel · LLaVA-1.5 | Rel. (%): 97.2 -> 97.9 | Helped |
| increase NC neighbor count from k=5 to k=10 (ablation) | PreSel · LLaVA-1.5 | Rel. (%): 97.9 -> 98.2 | Helped |
| increase NC neighbor count from k=10 to k=20 (ablation) | PreSel · LLaVA-1.5 | Rel. (%): 98.2 -> 97.4 | Hurt |
| reduce reference set size from 5% to 1% (ablation) | PreSel · LLaVA-1.5 | Rel. (%): 97.9 -> 97.7 | No clear effect |
Surprising findings
- Random selection outperforms many selection baselines that have access to all instructions, which the authors attribute to poor task distribution by those methods.
- IFD fails to improve with larger sampling ratios because it over-allocates budget to the LLaVA-v1 subset.
Limitations
- Uses reference instructions only for task-importance estimation; does not exploit them for selecting unlabeled images.
- Evaluated only on two existing VIT datasets whose instructions were used as generated instructions, not on newly generated data.
- Potential misuse: easier fine-tuning of open LVLMs for malicious purposes.
Experimental setup
Datasets: LLaVA-1.5 Vision-Flan
Benchmarks: VQAv2 ScienceQA-IMG TextVQA MME MMBench MMBench-CN SEED-Bench MM-Vet POPE
Metrics: average relative performance (%) accuracy
Baselines: Random TypiClust CLIP-Score EL2N Perplexity IFD Self-Filter COINCIDE
Builds on: LLaVA-1.5 DINOv2 TypiClust Self-Filter COINCIDE IFD
Closest papers by meaning
- VIGC: Visual Instruction Generation and Correction (AAAI 2024 · similarity 0.90)
- Focusing Where Vision Matters: Selective Training for Large Vision Language Models via Visual Information Gain (ICML 2026 · similarity 0.89)
- Learning to Instruct for Visual Instruction Tuning (NeurIPS 2025 · similarity 0.88)
- Adapt-$\infty$: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection (ICLR 2025 · similarity 0.87)
- Filter Like You Test: Data-Driven Data Filtering for CLIP Pretraining (NeurIPS 2025 · similarity 0.87)
- Concept-Aware Batch Sampling Improves Language-Image Pretraining (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.