Research map / Large language models / AI and society
No Free Delivery Service: Epistemic limits of passive data collection in complex social systems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretical paper arguing that train-test validation is provably invalid for passively collected, heavy-tailed data from complex social systems (recommenders, symbolic QA), illustrated with MovieLens rank-k matrix experiments.
Problem
Whether model validation via the train-test paradigm can be valid when data is passively collected from social systems with sampling bias and heavy-tailed degree distributions, as is common in recommender systems and LLM reasoning/QA.
Approach
Formalizes test validity as a property of an inference setting (assumptions, data, target distribution, distribution over possible worlds) and shows via k-core/k-connectivity of the sample graph that rank-k possible worlds are unidentified on passive data, so any risk estimator (including counterfactual/causal ones) is test-invalid under ontological parsimony; scaling and benchmarks are shown to b
Key results
- On MovieLens 100k, for 50% of unobserved entries the worst-case NAE across rank-50 possible worlds is above 77% of the maximum error.
- Book Crossing analysis: at rank 20 only 11.3% of users have large enough degrees for valid train-test inference; at rank 100 only 0.2%.
- For a rank-60 model, test-validity would hold for 67% of 'technicians' but only 14% of 'homemakers' in MovieLens 100k.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase model rank from 8 to 100 | Train-test paradigm on Book Crossing · Book Crossing users | fraction with valid train-test measures and inferences: 100% -> 0.2% | Hurt |
| Scale passive data collection to cover nodes under the Pareto distribution | Passive-data scaling · Pareto distribution, |X| = 10^7 | expected samples to increase k-core of random node: 2 \cdot 10^{21} samples | Hurt |
| Use a manually constructed benchmark to cover nodes with fewer than 100 observations | Manual benchmarks · Pareto distribution, |X| = 10^7 | expected nodes needing coverage: 9.9 \cdot 10^6 nodes | Hurt |
| Compare rank-50 possible worlds consistent with observed MovieLens data | MovieLens 100k rank-50 possible worlds · MovieLens 100k, unobserved entries | normalized absolute error (NAE): 23% to 49%; 77% | Mixed |
| Evaluate test validity by demographic group at Rank 60 | Train-test paradigm on MovieLens 100k · MovieLens 100k, technicians vs. homemakers | users for which test-validity holds: 67% vs 14% | Mixed |
| Increase assumed model complexity from Rank 8 to Rank 100 | Train-test paradigm on Book Crossing · Book Crossing users | fraction with valid train-test measures and inferences: 100% -> 0.2% | Hurt |
Surprising findings
- Test invalidity holds for any risk estimator, including counterfactual and causal estimators, not just empirical risk.
- Scaling is shown to be prohibitively inefficient: expected samples to reach a random node scale polynomially in domain size.
Limitations
- Results rely on the uniform distribution over possible worlds and the rank-k assumption; conclusions depend on these modeling choices.
- Experiments are limited to MovieLens 100k and a rank-based matrix setup; the LLM reasoning implications are argued rather than empirically tested.
- Scaling lower bounds are weak (only sampling a node once), and the k-core condition is necessary rather than sufficient.
Experimental setup
Datasets: MovieLens 100k Book Crossing
Benchmarks: MovieLens 100k
Metrics: normalized absolute error over unobserved entries fraction of users in k-core (test-validity coverage)
Builds on: No Free Lunch theorems matrix completion under non-uniform sampling Bregman divergence Pythagorean theorem
Closest papers by meaning
- AI Evaluation Should Require Standardized Item-Level Data Releases (NeurIPS 2026 · similarity 0.87)
- Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge (ICML 2025 · similarity 0.86)
- ACED-Bench: Evaluating How LLM Agents Acquire and Act on Causal Evidence (NeurIPS 2026 · similarity 0.85)
- Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor (NeurIPS 2025 · similarity 0.85)
- Adaptive auditing of AI systems with anytime-valid guarantees (NeurIPS 2026 · similarity 0.84)
- MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.