Research map / Vision-language and multimodal / Multimodal LLMs

FineVision: Open Data Is All You Need

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces FineVision, a 24M-sample open VLM training corpus unified from 200+ sources via a semi-automated, human-in-the-loop pipeline with SSCD-based dedup and decontamination, and shows a 460M SmolVLM trained on it beats Cauldron, Cambrian-7M and LLaVA-OneVision mixtures.

Problem

Public VLM training data is fragmented, inconsistently formatted, and contaminated with evaluation benchmark images, which hinders reproducible open VLM development.

Approach

Unify over 200 public multimodal sources into 185 subsets under one chat schema using LLM-assisted conversion with human review, apply SSCD-embedding near-duplicate removal and decontamination against 66 benchmarks, and release the corpus with per-turn quality scores and a unified GUI action schema.

Key results

• Models trained on FineVision average 49.6 over 11 benchmarks vs 38.9 (The Cauldron), 44.6 (Cambrian-7M), and 35.9 (LLaVA-OneVision).
• After decontamination FineVision drops only 1.6 pp versus 2.7-3.7 pp for baselines; its contamination rate is 1.02% vs 2.15-3.05%.
• On ScreenSpot-V2 after fine-tuning, FV-0.5B reaches 0.48 vs 0.41 for SmolVLM2-2.2B, a model about 4x larger.

Surprising findings

• Prompt-based quality-score thresholding did not improve downstream performance; training on the unfiltered data performed best.
• Merging multiple questions per image into multi-turn conversations did not improve performance in ablations.

Limitations

• Residual overlaps and contamination may persist; contamination flags are not definitive for samples.
• Long-context and multi-document reasoning remain challenging; GUI benchmarks not integrated in standard evaluation stack.
• Prompt-based quality scores were not effective filters, and licensing, privacy and bias audits are incomplete.

Experimental setup

Datasets: The Cauldron LLaVA-OneVision Cambrian-7M aguvis-stage-1 DenseFusion-1M Docmatix olmOCR-mix-0225 CoSyn_400k

Benchmarks: AI2D ChartQA DocVQA InfoVQA MME MMMU ScienceQA MMStar OCRBench TextVQA SEED-Bench ScreenSpot-V2

Metrics: accuracy average accuracy over 11 benchmarks contamination rate effective rank participation ratio

Baselines: The Cauldron LLaVA-OneVision Cambrian-7M

Builds on: SmolVLM nanoVLM The Cauldron Cambrian-1 LLaVA-OneVision

Closest papers by meaning

• Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.88)
• VHELM: A Holistic Evaluation of Vision Language Models (NeurIPS 2024 · similarity 0.88)
• Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.88)
• SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Models (CVPR 2025 · similarity 0.87)
• RedPajama: an Open Dataset for Training Large Language Models (NeurIPS 2024 · similarity 0.87)
• Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models (CVPR 2025 · similarity 0.87)