Research map / Vision-language and multimodal / Multimodal LLMs

MetaCaptioner: Towards Generalist Visual Captioning with Open-source Suites

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Interleaved image-text generation with MLLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CapFlow, a multi-agent, domain-routed captioning workflow built on open-source MLLMs, and uses it to synthesize 4.1M captions to fine-tune MetaCaptioner-8B, a generalist open-source captioner.

Problem

Open-source MLLMs produce generalist visual captions that lag far behind commercial models such as GPT-4.1 across diverse visual domains (charts, math, documents, medical, video), limiting data synthesis applications.

Approach

Decompose captioning into domain-routed hierarchical multi-agent workflows (perception, reasoning, tools, guideline agents followed by a summary agent) built on open-source Qwen2.5-VL-72B and Qwen2.5-72B, then distill the resulting high-quality captions, filtered by a reject-sampling judge, into an 8B captioner.

Key results

• CapFlow-72B reaches an average caption quality score of 2.33 vs. GPT-4.1's 2.35 on 250 samples judged by GPT-5, at about $0.14 vs $1.47 per sample (about 10.5% of the cost).
• With DeepSeek-R1-Distill-Qwen-32B as the LLM, MetaCaptioner-8B averages 55.3 vs 48.3 for OmniCaptioner-7B across visual-reasoning-with-LLM benchmarks.
• MetaCaptioner-8B averages 71.1 across direct multimodal benchmarks, with MathVerse 56.5 and MMMU 69.5, and +16.8 over GLM4.1V on ChartQA.

Surprising findings

• CapFlow's human-judged reasoning rigor scores exceed GPT-4.1's, while GPT-4.1 leads on GPT-5-judged factual accuracy.
• Adding MetaCaption captions to pre-training data improved average scores from 58.0 to 60.4 versus DenseFusion, with MetaCaption-450K reaching 61.8 average in the vanilla pre-training setting.

Limitations

• Caption quality is judged mainly by GPT-5 and Qwen2.5-VL-7B, which may bias results toward model-generated preferences.
• The annotation pipeline is costly (about 480 H200 GPU days) despite being cheaper than GPT-4.1 API calls.
• Reported in the paper as hallucination-prone, as noted in the ethics statement.

Experimental setup

Datasets: MetaCaption-4.1M (from 140+ open-source image and video datasets, e.g. SAM, LAION, ArxivQA, NovaChart, PMC-VQA, iNat) InternVL3.5 instruction data

Benchmarks: InfographicVQA ChartQA DocVQA AI2D MMBench MMMU MMVet MathVista MathVerse SEEDBench2 Plus MMStar VideoMME

Metrics: accuracy caption quality score (1-3 scale on five criteria) human preference score

Baselines: GPT-4.1 OmniCaptioner Qwen2.5-VL-7B InternVL3.5-8B GLM4.1V-9B Keye-VL-8B InternVL3-8B MiniCPM-V2.6-8B

Builds on: InternVL3.5 Qwen2.5-VL OmniCaptioner

Closest papers by meaning

• CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness (NeurIPS 2025 · similarity 0.88)
• FINECAPTION: Compositional Image Captioning Focusing on Wherever You Want at Any Granularity (CVPR 2025 · similarity 0.88)
• Enhancing Descriptive Captions with Visual Attributes for Multimodal Perception (CVPR 2026 · similarity 0.88)
• CapsFusion: Rethinking Image-Text Data at Scale (CVPR 2024 · similarity 0.87)
• ViPCap: Retrieval Text-Based Visual Prompts for Lightweight Image Captioning (AAAI 2025 · similarity 0.86)
• G-VEval: A Versatile Metric for Evaluating Image and Video Captions Using GPT-4o (AAAI 2025 · similarity 0.86)