Research map / Vision-language and multimodal / Multimodal LLMs
From Head to Tail: Towards Balanced Representation in Large Vision-Language Models through Adaptive Data Calibration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes long-tail imbalance in LVLM instruction-tuning data across tokens, objects, co-occurrences and interrogations, then rebalances head data and synthesizes tail data (ControlNet images plus LLM rewriting) to improve LLaVA 1.5 and ShareGPT4V without adding data.
Problem
Instruction-tuning data for large vision-language models is highly imbalanced (long-tail), with head concepts overrepresented and tail concepts underrepresented, which causes failures on tail concepts and degrades general benchmark performance.
Approach
Builds reverse-indexed entity distributions over four perspectives (tokens, objects, co-occurrences, interrogations), resamples redundant head instances with entity-based probabilities, and synthesizes tail data via ControlNet image editing and LLM-based text rewriting while keeping the data volume fixed.
Key results
- ADR on LLaVA 1.5 improves the average across eleven benchmarks by 2.28 points absolute and 4.36% relative, with no added training data.
- On ScienceQA-IMG tail@5% accuracy rises from 67.9 (LLaVA 1.5) to 70.1 with ADR, and overall accuracy from 69.3 to 70.2.
- Ours-DR and Ours match or beat EL2N, Perplexity, CLIP Score and COINCIDE on GQA, SEED, SEEDv2, POPE and MMB in Table 4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply ADR data rebalancing and synthesis | LLaVA 1.5 · 11 comprehensive benchmarks | average performance: relative improvement of 4.36% | Helped |
| Apply ADR data rebalancing and synthesis | LLaVA 1.5 · VQAOK | score: 53.2 -> 57.4 | Helped |
| Apply ADR data rebalancing and synthesis | LLaVA 1.5 · SEED2 | score: 48.7 -> 57.4 | Helped |
| Apply ADR data rebalancing and synthesis | LLaVA 1.5 · MMEP | score: 1510.7 -> 1512.8 | Helped |
| Apply ADR data rebalancing and synthesis | LLaVA 1.5 · ScienceQA tail @5% | accuracy: 67.9 -> 70.1 | Helped |
| Apply ADR data rebalancing and synthesis | LLaVA 1.5 · ScienceQA overall | accuracy: 69.3 -> 70.2 | Helped |
| Use all four entity perspectives for rebalancing (ablation) | ADR data rebalancing · Comprehensive benchmarks | average score: 60.6 | Helped |
| Use full visual synthesis for selected tail instances (ablation) | ADR data synthesis · Comprehensive benchmarks | average score: 61.1 | Helped |
| Keep conversation text unchanged during visual synthesis (ablation) | ADR data synthesis · Comprehensive benchmarks | average score: 59.8 | Mixed |
| Use language data synthesis (ablation) | ADR data synthesis · Comprehensive benchmarks | average score: 59.9 | Mixed |
| Select with Token and Interrogation and rewrite conversations (ablation) | ADR data synthesis · Comprehensive benchmarks | average score: 60.3 | Mixed |
| Use simple resampling for selected tail data (ablation) | ADR data synthesis · Comprehensive benchmarks | average score: 59.9 | Mixed |
Surprising findings
- Tail 50% of failed cases on MME and POPE cover over 99% of entities yet account for 68.65% of training instances on average, showing failures concentrate in tail concepts.
Limitations
- Synthesis and rebalancing evaluated only on LLaVA 1.5 and ShareGPT4V instruction tuning data; generality to other LVLMs is not shown.
- COINCIDE results are taken from its paper because code was not released, so comparison is not controlled.
- Tail analysis is limited to ScienceQA and VQAv2.
Experimental setup
Datasets: LLaVA 1.5 instruction tuning data (665K) ShareGPT4V instruction tuning data MME POPE ScienceQA-IMG VQAv2
Benchmarks: VQAv2 OK-VQA TextVQA GQA MMBench MME MMStar SEED-Bench SEED-Bench2 ScienceQA MMMU Q-Bench2
Metrics: accuracy average score relative improvement
Baselines: LLaVA 1.5 ShareGPT4V EL2N Perplexity CLIP Score COINCIDE
Builds on: LLaVA 1.5 ShareGPT4V ControlNet GroundingDINO COINCIDE
Closest papers by meaning
- A-VL: Adaptive Attention for Large Vision-Language Models (AAAI 2025 · similarity 0.90)
- Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models (ICLR 2024 · similarity 0.90)
- VladVA: Discriminative Fine-tuning of LVLMs (CVPR 2025 · similarity 0.90)
- The Neglected Tails in Vision-Language Models (CVPR 2024 · similarity 0.90)
- Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation (ICML 2026 · similarity 0.90)
- Enhancing Large Vision Language Models with Self-Training on Image Comprehension (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.