Research map / Vision-language and multimodal / Human and embodied modeling
OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds OmniCorpus, a 10-billion-level bilingual image-text interleaved corpus (8.6B images, 1,696B text tokens) from Common Crawl, Chinese websites, and video platforms, using a scalable data engine with human-feedback text filters.
Problem
Existing image-text interleaved datasets are limited in scale and source diversity (mostly English Common Crawl), which restricts training of multimodal large language models.
Approach
A unified streaming data format that represents HTML documents, image-text pairs, and video storyboards as interleaved sequences, produced by a parallel, throughput-balanced data engine with human-feedback text filters and pre-deduplication to scale to billions of images.
Key results
- OmniCorpus has 8.6B images, 1,696B text tokens and 2.2B documents, about 1.7x LAION-5B's images and 12.5x its text.
- OmniCorpus-CC-pretrained baseline gets the best few-shot average among MMC4, MMC4-Core, OBELICS and OmniCorpus-CC subsets of 1M documents (e.g., 58.7 at 4-shot vs OBELICS 58.2).
- A 7B model trained on LAION plus OmniCorpus-CC outperforms the 9B OpenFlamingo and IDEFICS on most benchmarks, e.g., VQAv2 64.5 vs 54.8 and 55.4 at 4-shot.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on the 200M curated subset instead of the 988M subset (ablation) | LLaVA-1.5 baseline trained on OmniCorpus-CC · Four-benchmark average, 4-shot | average score: 54.5 -> 58.7 | Helped |
| Use progressively smaller curated OmniCorpus-CC subsets (ablation) | LLaVA-1.5 baseline · Four-benchmark average | average score: 28.2; 28.9; 28.3; 24.6; 25.5; 28.4 (0-shot) | Mixed |
| Pre-train exclusively on OmniCorpus-YT video-derived documents | LLaVA-1.5 baseline · VQA and image captioning benchmarks | benchmark performance | Mixed |
| Pre-train on OmniCorpus-CW Chinese documents before LLaVA-665K fine-tuning | MLLM · MMBench-CN | score: 59.8 -> 62.5 (+2.7) | Helped |
| Pre-train on OmniCorpus-CW Chinese documents before LLaVA-665K fine-tuning | MLLM · CMMMU | score: 23.6 -> 24.9 (+1.3) | Helped |
| Use the natural image-position strategy rather than retrieval-based placement (ablation) | Fully autoregressive architecture · OmniCorpus-CC image-text interleaved pre-training | performance | Helped |
| Use retrieval-based image placement rather than natural layout (ablation) | Cross-attention architecture · OmniCorpus-CC image-text interleaved pre-training | performance | Helped |
Surprising findings
- The natural layout ordering works better for fully autoregressive architectures, while retrieval-based (CLIP-similarity) ordering works better for cross-attention architectures.
- Over-filtering the data (down to 200M from 600M documents) did not help further, which the authors attribute to data homogenization.
Limitations
- Current filtering offers limited improvements to model performance, and the factors that make data beneficial are not explored.
- Dataset contains URLs rather than media, so external resources may disappear over time; may contain biased or harmful content despite filtering.
- No error bars reported for experiments.
Experimental setup
Datasets: OmniCorpus-CC OmniCorpus-CW OmniCorpus-YT MMC4 MMC4-Core OBELICS LAION-5B LLaVA-665K YT-Temporal-1B HD-VILA-100M HowTo100M InternVid
Benchmarks: OKVQA TextVQA COCO Flickr30k VQAv2 VizWiz MMBench-CN CMMMU MMLU
Metrics: accuracy CIDEr zero-shot and few-shot scores
Baselines: MMC4 MMC4-Core OBELICS OpenFlamingo-9B IDEFICS-9B Emu-14B
Builds on: Trafilatura MMC4 OBELICS LLaVA-1.5 OpenFlamingo
Closest papers by meaning
- OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation (CVPR 2025 · similarity 0.91)
- CoMM: A Coherent Interleaved Image-Text Dataset for Multimodal Understanding and Generation (CVPR 2025 · similarity 0.91)
- A High Quality Dataset and Reliable Evaluation for Interleaved Image-Text Generation (ICLR 2026 · similarity 0.90)
- MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens (NeurIPS 2024 · similarity 0.89)
- TRINS: Towards Multimodal Language Models that Can Read (CVPR 2024 · similarity 0.88)
- CapsFusion: Rethinking Image-Text Data at Scale (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.