Research map / Vision-language and multimodal / Multimodal LLMs
Demystifying Supervision Data Generalization in Multimodal LMs
Xuan Qi, Luxi He, Dan Roth, Xingyu Fu
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies whether intuitive similarity between supervision datasets and target benchmarks reliably predicts MLLM performance gains, and how to select training data accordingly.
Problem
Selecting supervision data for multimodal LLMs by intuitive task similarity (e.g., text-rich vs. vision-centric) may not reliably translate into better test benchmark performance.
Approach
Treats the question empirically: asks whether the effect of one supervision dataset on a test benchmark can be predicted, rather than relying on intuitive similarity between datasets and targets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| include multimodal perplexity in the influence metric (ablation) | DataProphet · data influence ranking | Kendall's tau: 0.487 -> 0.860 | Improved |
| include image similarity (ablation) | DataProphet · data influence ranking | Kendall's tau: 0.625 -> 0.860 | Improved |
| include question diversity (silhouette + entropy) (ablation) | DataProphet · data influence ranking | Kendall's tau: 0.659 -> 0.860 | Improved |
| include question similarity (ablation) | DataProphet · data influence ranking | Kendall's tau: 0.778 -> 0.860 | Improved |
| include answer similarity (ablation) | DataProphet · data influence ranking | Kendall's tau: 0.810 -> 0.860 | Improved |
| add answer length to the metric (ablation) | DataProphet · data influence ranking | Kendall's tau: -0.15 | Hurt |
| encode question and answer jointly instead of separately (ablation) | DataProphet similarity · data influence ranking | Kendall's tau: -6% | Hurt |
| use DataProphet reweighting instead of uniform mixture | InternVL3-2B SFT · 14 benchmarks (real data) | average score: 67.6 -> 71.0 | Improved |
| use DataProphet instead of ICONS (training-based) for real data | InternVL3-2B SFT · 14 benchmarks (real data) | average score: 69.6 -> 71.0 | Improved |
| use DataProphet instead of Oracle reweighting | InternVL3-2B SFT · 14 benchmarks (real data) | average score: 70.8 -> 71.0 | No clear effect |
| use DataProphet selection on synthetic pool instead of uniform | InternVL3-2B SFT · 14 benchmarks (synthetic data) | average score: 55.1 -> 62.0 | Improved |
| use DataProphet allocation for RL post-training data | Qwen2.5VL-3B-Instruct GRPO · 14 benchmarks | average score: 58.3 -> 59.5 (real); 56.4 -> 57.7 (synthetic) | Improved |
Surprising findings
- Intuitive similarity between supervision data and target task (e.g., text-rich vs. vision-centric) is not a reliable predictor of test benchmark gains, per the abstract's framing.
Closest papers by meaning
- MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning (ICLR 2025 · similarity 0.88)
- On Path to Multimodal Generalist: General-Level and General-Bench (ICML 2025 · similarity 0.88)
- NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints (NeurIPS 2025 · similarity 0.88)
- OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging (ICLR 2026 · similarity 0.88)
- Learn from Downstream and Be Yourself in Multimodal Large Language Models Fine-Tuning (ICML 2025 · similarity 0.88)
- MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
