Research map / Large language models / AI and society
Consent in Crisis: The Rapid Decline of the AI Data Commons
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Audits consent signals (robots.txt and Terms of Service) across web domains underlying C4, RefinedWeb, and Dolma, tracking how AI-related restrictions grew from 2016 to 2024.
Problem
Web domains underlying AI training corpora increasingly restrict crawling and AI use through robots.txt and Terms of Service, but no longitudinal audit existed of these consent signals and how they affect the data commons.
Approach
Traces provenance beneath datasets to the source web domains, combining Wayback Machine longitudinal snapshots of robots.txt and ToS with human and GPT-4o annotation of domain characteristics, then forecasts restriction trends with SARIMA and compares web sources with real ChatGPT usage in WildChat.
Key results
- About 5-7% of tokens across C4, RefinedWeb, and Dolma were robots.txt-restricted in April 2024, versus about 1% in mid-2023.
- For Terms of Service, 45-55% of tokens in these corpora carry some form of data use restriction.
- OpenAI crawlers are restricted for 25.9% of tokens in the top C4 head, the highest among AI developers audited.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Restrictions on crawling increased from mid-2023 to April 2024 | C4, RefinedWeb, and Dolma full corpora · Full corpora | tokens fully restricted by robots.txt: ~1% -> 5-7% | Hurt |
| Restrictions on crawling increased from mid-2023 to April 2024 | C4, RefinedWeb, and Dolma head domains · Head domains | tokens fully restricted by robots.txt: <3% -> 20-33% | Hurt |
| Terms of Service crawling restrictions increased over the last year | C4, RefinedWeb, and Dolma · Full corpora | Terms of Service crawling restrictions: 26-53% relative increase | Hurt |
| Restricting crawlers in robots.txt | C4 head domains · C4 head domains | tokens restricted by crawler: OpenAI 25.9%; Anthropic and Common Crawl 13.3%; Google AI 9.8%; Cohere 4.9%; Meta 4.1%; Internet Archive 3.2%; Google Se | Mixed |
| Concentrating on the top 2000 domains rather than a random sample | Web source audit · Audited domains | restrictive robots.txt incidence (%): 3.4 -> 26.5 | Hurt |
| Concentrating on the top 2000 domains rather than a random sample | Web source audit · Audited domains | restrictive Terms incidence (%): 15.7 -> 51.2 | Hurt |
| Concentrating on the top 2000 domains rather than a random sample | Web source audit · Audited domains | advertisement incidence (%): 5.4 -> 53.2 | Mixed |
| Concentrating on the top 2000 domains rather than a random sample | Web source audit · Audited domains | video modality incidence (%): 18.9 -> 58.7 | Helped |
| Concentrating on the top 2000 domains rather than a random sample | Web source audit · Audited domains | audio modality incidence (%): 3.4 -> 41.8 | Helped |
| Concentrating on the top 2000 domains rather than a random sample | Web source audit · Audited domains | news/periodicals incidence (%): 5.3 -> 50.0 | Helped |
| Concentrating on the top 2000 domains rather than a random sample | Web source audit · Audited domains | organization/personal website incidence (%): 71.2 -> 12.7 | Hurt |
| Comparing real user requests with content prevalent in web-derived training sources | ChatGPT use and public web data · WildChat conversations | creative composition requests: over 30% | Mixed |
Surprising findings
- News domains saw 45% of tokens fully restricted by robots.txt in the head sample by April 2024, versus 3% in 2023.
- Fewer than 1% of ChatGPT queries in WildChat relate to news, despite news comprising nearly 40% of head tokens in C4.
Limitations
- Wayback Machine gaps cause non-smooth temporal trends and exclusion of unretrievable domains.
- Restrictions are not legally enforceable and it is unclear whether crawlers respect them; the paper makes no assertion on consent implied by omission.
- WildChat is a sample from a technical HuggingFace Spaces proxy and may not represent general ChatGPT usage.
Experimental setup
Datasets: C4 RefinedWeb Dolma WildChat
Benchmarks: WildChat
Metrics: percent of tokens restricted incidence rates of domain features conditional restriction probabilities SARIMA forecasts
Baselines: Googlebot robots.txt restrictions
Builds on: C4 RefinedWeb Dolma SARIMA
Closest papers by meaning
- Bridging the Data Provenance Gap Across Text, Speech, and Video (ICLR 2025 · similarity 0.85)
- AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agents (NeurIPS 2025 · similarity 0.84)
- Stop DDoS Attacking the Research Community with AI-Generated Survey Papers (NeurIPS 2025 · similarity 0.83)
- Position: A Safe Harbor for AI Evaluation and Red Teaming (ICML 2024 · similarity 0.83)
- On the Consideration of AI Openness: Can Good Intent Be Abused? (AAAI 2025 · similarity 0.83)
- Position: Data Authenticity, Consent, & Provenance for AI are all broken: what will it take to fix them? (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.