Research map / Vision-language and multimodal / Multimodal LLMs
MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MINT-1T, an open-source trillion-token multimodal interleaved pre-training dataset built from CommonCrawl HTML, PDFs, and ArXiv, and shows LMMs trained on it roughly match OBELICS.
Problem
Open-source multimodal interleaved pre-training datasets are small (OBELICS: 115B tokens) and draw only from HTML, limiting open LMM development relative to closed models.
Approach
Scale interleaved image-text data to one trillion text tokens by adding PDF (CommonCrawl WAT) and ArXiv LaTeX sources alongside an extended OBELICS-style HTML pipeline, with rule-based filtering and deduplication rather than model-based text filters.
Key results
- MINT-1T-trained 4.6B model reaches 62.3 average 4-shot and 64.6 8-shot ICL vs OBELICS 61.2 and 63.3 (XGen-MM setup, 10B tokens).
- On MMMU, the MINT-1T model scores 41.3 vs OBELICS 37.6, but on Mantis-Eval it scores 43.3 vs OBELICS 44.2.
- With the Idefics2 architecture, MINT-1T (HTML) outperforms OBELICS on average (66.9 vs 66.7 at 8-shot, 65.0 vs 64.3 at 4-shot).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on MINT-1T instead of OBELICS | Phi-3 mini LMM · In-context learning, 4-shot | average performance: 61.2 ± 0.28 -> 62.3 ± 0.13 | Helped |
| Train on MINT-1T instead of OBELICS | Phi-3 mini LMM · In-context learning, 8-shot | average performance: 63.3 ± 0.24 -> 64.6 ± 0.14 | Helped |
| Train on MINT-1T instead of MINT-1T (HTML) | Phi-3 mini LMM · In-context learning, 4-shot | average performance: 60.9 ± 0.27 -> 62.3 ± 0.13 | Helped |
| Train on MINT-1T instead of MINT-1T (HTML) | Phi-3 mini LMM · In-context learning, 8-shot | average performance: 63.0 ± 0.28 -> 64.6 ± 0.14 | Helped |
| Train on MINT-1T instead of OBELICS | Phi-3 mini LMM · MMMU validation | performance: 37.6 -> 41.3 | Helped |
| Train on MINT-1T instead of OBELICS | Phi-3 mini LMM · Mantis-Eval | performance: 44.2 -> 43.3 | Hurt |
| Train on MINT-1T (HTML) instead of OBELICS | Phi-3 mini LMM · In-context learning, 4-shot | average performance: 61.2 ± 0.28 -> 60.9 ± 0.27 | Hurt |
| Train on MINT-1T (HTML) instead of OBELICS | Idefics2 · In-context learning, 4-shot | average performance: 64.3 ± 0.24 -> 65.0 ± 0.12 | Helped |
| Train on MINT-1T (HTML) instead of OBELICS | Idefics2 · In-context learning, 8-shot | average performance: 66.7 ± 0.20 -> 66.9 ± 0.19 | Helped |
| Use MINT-1T rather than MINT-1T (HTML) | Phi-3 mini LMM · Mantis-Eval | performance: 41.5 -> 43.3 | Helped |
| Use MINT-1T rather than MINT-1T (HTML) | Phi-3 mini LMM · MMMU validation | performance: 35.2 -> 41.3 | Helped |
Surprising findings
- The HTML-only subset of MINT-1T underperforms OBELICS on average under the XGen-MM architecture but outperforms it under the Idefics2 architecture.
- MINT-1T trained models do not beat OBELICS on Mantis-Eval, despite gains on MMMU.
Limitations
- PDF reading order and image detection rely on PyMuPDF heuristics and sometimes extract incorrect order or miss images.
- Dataset is over 300TB and does not host images; relies on external URLs subject to link rot.
- Safety filtering (NSFW, PII) is imperfect; authors acknowledge potential harmful content remains.
Experimental setup
Datasets: MINT-1T OBELICS internal image-text caption datasets CommonCrawl WARC CommonCrawl WAT ArXiv LaTeX sources
Benchmarks: COCO TextCaps OKVQA TextVQA VQAv2 VizWiz MMMU Mantis-Eval
Metrics: CIDEr VQA accuracy mean accuracy over trials
Baselines: OBELICS MINT-1T (HTML)
Builds on: OBELICS XGen-MM Dolma OpenFlamingo Multimodal-C4
Closest papers by meaning
- OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text (ICLR 2025 · similarity 0.89)
- DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models (CVPR 2025 · similarity 0.87)
- Data-Juicer Sandbox: A Feedback-Driven Suite for Multimodal Data-Model Co-development (ICML 2025 · similarity 0.87)
- OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation (CVPR 2025 · similarity 0.87)
- MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks (ICLR 2025 · similarity 0.86)
- A High Quality Dataset and Reliable Evaluation for Interleaved Image-Text Generation (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.