Research map / Vision-language and multimodal / Multimodal LLMs
TaskGalaxy: Scaling Multi-modal Instruction Fine-tuning with Tens of Thousands Vision Task Types
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds TaskGalaxy, a ~413k-sample multimodal instruction tuning dataset spanning ~19k hierarchical task types, generated by GPT-4o with CLIP matching and open-source model filtering, and shows gains when added to LLaVA-v1.5 and InternVL-Chat-v1.0 SFT.
Problem
Existing multimodal instruction fine-tuning datasets have limited task-type diversity (around 200 types) because manual task labeling is labor-intensive, which limits generalization of visual language models.
Approach
Expand a small set of seed visual task types hierarchically with GPT-4o, match task types to open-source images via CLIP similarity followed by GPT-4o filtering, generate QA pairs with GPT-4o, and screen them with three open-source multimodal judges by majority-style score aggregation.
Key results
- Adding TaskGalaxy to LLaVA-v1.5-7B SFT raises the average over the 15 non-MME benchmarks from 44.46 to 48.96 (+4.5) and MME perception from 1506 to 1533.
- LLaVA-v1.5-13B gains 68 points on MME (1532 to 1600) with TaskGalaxy, and its non-MME average rises from 45.21 to 49.04.
- InternVL-Chat-v1.0-13B non-MME average rises from 49.90 to 53.54 (+3.64) after adding TaskGalaxy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add TaskGalaxy to SFT data | LLaVA-v1.5-7B · 15 benchmarks, excluding MME | average score: 44.46 -> 48.96 | Helped |
| Add TaskGalaxy to SFT data | LLaVA-v1.5-13B · 15 benchmarks, excluding MME | average score: 45.21 -> 49.04 | Helped |
| Add TaskGalaxy to SFT data | InternVL-Chat-v1.0-7B · 15 benchmarks, excluding MME | average score: 47.79 -> 50.79 | Helped |
| Add TaskGalaxy to SFT data | InternVL-Chat-v1.0-13B · 15 benchmarks, excluding MME | average score: 49.90 -> 53.54 | Helped |
| Add TaskGalaxy to SFT data | LLaVA-v1.5-13B · MME | perception score: 1532 -> 1600 | Helped |
| Increase task-type count while holding sample size at 100k (ablation) | LLaVA-v1.5-7B · LLaVA-in-the-wild, ChartQA, AI2D, Q-Bench, MMMU, average across 15 benchmarks, and MME | performance | Helped |
| Increase sample volume across all 19,227 task types (ablation) | LLaVA-v1.5-7B · MMVet, ChartQA, AI2D, Q-Bench, and average across 15 benchmarks | performance | Helped |
| Increase sample volume beyond the reported peak (ablation) | LLaVA-v1.5-7B · LLaVA-in-the-wild | performance: 281k samples | Mixed |
Surprising findings
- Adding TaskGalaxy yields a 68-point MME gain for LLaVA-v1.5-13B, far larger than gains on other benchmarks.
- CLIP-based task-image matching often produces illusory matches (e.g., food images paired with menu OCR tasks), requiring GPT-4o re-filtering.
Limitations
- Relies on GPT-4o for task expansion, matching, and QA generation, so inherits its hallucinations and biases.
- CLIP-based initial matching produces illusory task-image pairs that need correction.
- Gains are from single-run comparisons with no reported variance.
Experimental setup
Datasets: ALLaVA Visual Genome MathV360K ShareGPT4V COCO LLaVA-Mix-665K LCS-558K CC3M
Benchmarks: MME MMBench MMBench-CN POPE LLaVA-in-the-wild MMVet TextVQA ScienceQA MathVista ChartQA AI2D Q-Bench
Metrics: accuracy perception score average score
Baselines: LLaVA-v1.5 InternVL-Chat-v1.0 Vision-Flan VisionLLM v2
Builds on: LLaVA InternVL-Chat Vision-Flan CLIP
Closest papers by meaning
- Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment (CVPR 2025 · similarity 0.91)
- VIGC: Visual Instruction Generation and Correction (AAAI 2024 · similarity 0.89)
- CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models (CVPR 2026 · similarity 0.89)
- Adapt-$\infty$: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection (ICLR 2025 · similarity 0.89)
- OddGridBench: Exposing the Lack of Fine-Grained Visual Discrepancy Sensitivity in Multimodal Large Language Models (CVPR 2026 · similarity 0.89)
- Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.