Research map / Vision-language and multimodal / Multimodal LLMs

Data Selection for Fine-tuning Vision Language Models via Cross Modal Alignment Trajectories

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Visual instruction tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes XMAS, a data selection method for visual instruction tuning that clusters examples by the trajectories of top singular values of their cross-modal attention matrices from a small proxy LVLM and samples stable examples in a balanced way.

Problem

Data-efficient instruction tuning of large vision-language models: existing selection methods fail to beat random selection at various subset sizes, and gradient-based selection is too expensive for LVLMs.

Approach

Theoretically bounds pairwise gradient distance by distance between cross-modal attention matrices (for small-curvature fine-tuning across checkpoints), then clusters examples by their cross-modal alignment score trajectories from a proxy LVLM and samples a balanced subset of the most stable (lowest instability) examples from each cluster.

Key results

• On LLaVA 665k, XMAS reaches 100.0 ARP at 50% of data, versus 99.8 for HL and 99.6 for MP, roughly 30% more data reduction than the best baseline.
• On Vision-Flan, XMAS achieves 100.2 ARP at 15% of data (6.7x reduction) and outperforms random selection by about 2% at lower budgets (96.1 vs 92.7 at 5%).
• XMAS yields a 1.2x training speedup including selection time on LLaVA 665k at 50% data.

Surprising findings

• Single-metric baselines such as Middle Perplexity and Highest Learnability fall below random selection at 10-30% budgets on LLaVA 665k.
• COINCIDE is only 1.1x slower than full training because it can discard only 10% of data.

Limitations

• Theoretical guarantees rely on single-layer, single-head transformer analysis and strong assumptions (bounded norms, small gain, small curvature).
• Proxy-target trajectory similarity is validated empirically only for one proxy/target pair.
• Experiments limited to LLaVA-1.5 target models and two instruction-tuning datasets.

Experimental setup

Datasets: LLaVA 665k Vision-Flan

Benchmarks: POPE TextVQA MME-Perception ScienceQA VizWiz MMBench LLaVA-Bench MM-Vet VQAv2 GQA

Metrics: average relative performance (ARP) relative performance to full data

Baselines: Random Middle Perplexity Highest Learnability EL2N Self-Sup CLIP-Score SemDeDup D2 Pruning Self-Filter COINCIDE

Builds on: LESS COINCIDE S2L LLaVA-1.5

Closest papers by meaning

• Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.90)
• Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradients (CVPR 2026 · similarity 0.90)
• VladVA: Discriminative Fine-tuning of LVLMs (CVPR 2025 · similarity 0.89)
• Vision-Language Model Selection and Reuse for Downstream Adaptation (ICML 2025 · similarity 0.89)
• A Comprehensive Information-Decomposition Analysis of Large Vision-Language Models (ICLR 2026 · similarity 0.89)
• Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation (ICML 2026 · similarity 0.89)