Research map / Video / Video generation
Bridging the Data Provenance Gap Across Text, Speech, and Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Manual longitudinal audit of ~3900 popular text, speech, and video datasets (1990-2024) tracing sourcing, license and source-terms restrictions, and geographic/linguistic representation, with the full audit released.
Problem
Public multimodal AI training datasets beyond text are poorly documented, with unclear data sourcing, usage restrictions, and geographic/linguistic representation, creating legal, privacy, and fairness risks for developers.
Approach
Extends the Data Provenance Initiative audit methodology across text, speech, and video, adding a taxonomy for source terms (Unrestricted, Unspecified, Source Closed, Model Closed) alongside dataset licenses and measuring representation with Gini coefficients over time.
Key results
- Less than 33% of datasets are restrictively licensed, yet over 80% of source content in widely used text, speech, and video datasets carries non-commercial restrictions.
- Geographic and linguistic representation has not significantly improved since 2013; Gini coefficients for creator geography are 0.92 (text), 0.86 (speech), 0.74 (video).
- Web, social media (YouTube supplies 71% of video and 69% of speech data) and synthetic sources dominate since 2019.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Web-crawled, social-media, and synthetic sources increasingly replace curated sources in multimodal datasets | Public text, speech, and video dataset ecosystem · Speech and video | source prevalence: Nearly 1M hours each of Speech and Video data from this source; less than 100K hours | Mixed |
| Use YouTube as a data source | Public speech and video datasets · Speech and video | share of data from YouTube: 71% of Video data and 69% of Speech data | Mixed |
| Derive datasets from sources with non-commercial restrictions | Text dataset content · Text | content with source restrictions: 99.8% | Hurt |
| Rely on dataset licenses without tracing source terms | Text, speech, and video datasets · Text, speech, and video | content with inconsistent license and source restrictions: 79% of tokens in text datasets, 55% of speech hours, and 65% of video hours | Hurt |
| Increase the number of represented countries over time | Dataset creator organizations · Text, speech, and video | geographical representation equality | No clear effect |
| Add multilingual datasets over time | Text and speech dataset ecosystem · Text and speech | language distribution equality | Mixed |
| Release large-scale multilingual projects such as xP3x and Common Voice | Text language representation · Text | language distribution equality: introduced in 2019 | Helped |
| Use synthetic rather than natural text datasets | Text datasets · Text | share containing classification tasks: 88.1% of natural datasets ... 66.3% of synthetic datasets | Mixed |
Surprising findings
- Dataset licenses conflict with source terms for over 55% of content, with 79% of text tokens showing such inconsistency.
- Commercially licensed datasets are largely restricted by their sources' terms, e.g., 99.8% of text quantities carry some non-commercial source restriction.
Limitations
- Creator affiliation information is partial, and privacy reasons limit transparency.
- Proprietary datasets may fill representation gaps not captured by the audit.
- Dataset selection is per-modality and may not represent each modality's full ecosystem; images, 3D, tabular and other modalities excluded.
Experimental setup
Datasets: HuggingFace Datasets SFT/DPO text datasets ASR speech datasets (OpenSLR, HuggingFace) video datasets (HuggingFace, PapersWithCode)
Metrics: dataset count token or hour size Gini coefficient license and terms restriction share
Builds on: Data Provenance Collection
Closest papers by meaning
- The State of Data Curation at NeurIPS: An Assessment of Dataset Development Practices in the Datasets and Benchmarks Track (NeurIPS 2024 · similarity 0.86)
- RedPajama: an Open Dataset for Training Large Language Models (NeurIPS 2024 · similarity 0.86)
- Consent in Crisis: The Rapid Decline of the AI Data Commons (NeurIPS 2024 · similarity 0.85)
- The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text (NeurIPS 2025 · similarity 0.85)
- TRACER: Verifiable Generative Provenance for Multimodal Tool-Using Agents (NeurIPS 2026 · similarity 0.85)
- A Systematic Review of NeurIPS Dataset Management Practices (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.