Research map / Large language models / AI and society
Data commons and human baselines
Measure AI's societal footprint, data consent, and rigor of human baselines.
159 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in AI and society, keyword cluster “human · systems · argue”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
29 papers in 2023–24 and 104 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 12.8% to 20.4%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · News Verification by Exploiting Conflicting Social Viewpoints in Microblogs (AAAI 2016 · 371 citations)
- 2017 · Long-Term Trends in the Public Perception of Artificial Intelligence (AAAI 2017 · 455 citations)
- 2018 · A Voting-Based System for Ethical Decision Making (AAAI 2018 · 144 citations)
- 2019 · SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (NeurIPS 2019 · 984 citations)
- 2020 · Predictive Engagement: An Efficient Metric for Automatic Evaluation of Open-Domain Dialogue Systems (AAAI 2020 · 45 citations)
- 2021 · CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation (NeurIPS 2021 · 408 citations)
- 2023 · Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies (ICML 2023 · 122 citations)
- 2024 · Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews (ICML 2024 · 70 citations)
- 2025 · The Illusion of Empathy: How AI Chatbots Shape Conversation Perception (AAAI 2025 · 19 citations)
- 2026 · Who’s in Charge? Disempowerment Patterns in Real-World LLM Usage (ICML 2026 · 2 citations)
Earliest papers here
- News Verification by Exploiting Conflicting Social Viewpoints in Microblogs (AAAI 2016 · 371 citations)
- Social Role-Aware Emotion Contagion in Image Social Networks (AAAI 2016 · 42 citations)
- Predicting Readers' Sarcasm Understandability by Modeling Gaze Behavior (AAAI 2016 · 39 citations)
What has been tried
69 papers here report 609 measured changes (a component added, removed or swapped, compared with a baseline): helped 61%, no clear effect 8%, hurt 23%, mixed 9%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Use adjectives rather than adverbs as the token vocabulary | Semi-synthetic validation · prediction error | Helped | Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews (ICML 2024 · 70 citations) | |
| remove suggestion features | Copilot acceptance prediction, session split · AU-ROC | 0.795 -> 0.631 | Hurt | When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming (AAAI 2024 · 19 citations) |
| Use full Gopher filters instead of only Gopher natural-language filters | RPv2 full · average benchmark score | 47.9 -> 50.0 | Helped | RedPajama: an Open Dataset for Training Large Language Models (NeurIPS 2024 · 19 citations) |
| Add noise symbols to verbal stimuli | Verbal n-back · working memory capacity | Hurt | Working Memory Capacity of ChatGPT: An Empirical Study (AAAI 2024 · 8 citations) | |
| Train the Prototypical Network on ToM-in-AMC rather than TVSG | ToM-in-AMC Test · accuracy | 46.4 -> 53.2 | Helped | Few-Shot Character Understanding in Movies as an Assessment to Meta-Learning of Theory-of-Mind (ICML 2024 · 4 citations) |
| Train the probe on Roleplaying instead of Instructed-Pairs | Insider Trading Report and Sandbagging · AUROC for deceptive versus honest responses | Mixed | Detecting Strategic Deception with Linear Probes (ICML 2025 · 1 citation) | |
| Increase paper-writing agents from 1 to 2 | Paper writing · similarity | 49.0 -> 52.7 | Helped | ResearchTown: Simulator of Human Research Community (ICML 2025 · 1 citation) |
| Include demographic information in the prompt | DIVE human-rating comparison · Kendall's Tau | 0.024 -> 0.23 | Helped | Whose View of Safety? A Deep DIVE Dataset for Pluralistic Alignment of Text-to-Image Models (NeurIPS 2025 · 1 citation) |
| Add timestamped PoseC3D action labels | MimeQA subset, overall · accuracy | 16.84% to 16.16% | Hurt | MimeQA: Towards Socially-Intelligent Nonverbal Foundation Models (NeurIPS 2025 · 1 citation) |
| add chain-of-thought prompting instead of QA prompting | RPS, IBS, and IPD, single-action partners · functional regret per step | 0.444 ± 0.107 -> 0.213 ± 0.015; 1.391 ± 0.283 -> 2.475 ± 0.208; 0.996 ± 0.133 -> 0.892 ± 0.210 | Mixed | Position: Theory of Mind Benchmarks are Broken for Large Language Models (ICML 2025 · 0 citations) |
| Use multiturn-aware rewards with forward sampling instead of immediate rewards | MediumDocEdit-Chat · BLEU | Helped | CollabLLM: From Passive Responders to Active Collaborators (ICML 2025 · 0 citations) | |
| Apply supervised fine-tuning on PANORAMA | PAR4PC · score | 13.45 -> 77.98 | Helped | PANORAMA: A Dataset and Benchmarks Capturing Decision Trails and Rationales in Patent Examination (NeurIPS 2025 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 64 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- MNLI · 2 of 64 papers
- SST2 · 2 of 64 papers
- WildChat · 2 of 64 papers
Baselines
- GPT-4o · 9 of 64 papers
- Llama-3.1-8B-Instruct · 4 of 64 papers
- Gemini 3.1 Pro · 4 of 64 papers
- DeepSeek-V3 · 3 of 64 papers
- Gemini-2.5-Pro · 3 of 64 papers
- Claude Haiku 4.5 · 3 of 64 papers
- Cohen's kappa · 2 of 64 papers
- GPT-3.5-turbo · 2 of 64 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 133 papers; from OpenAlex and the papers' affiliations).
- Stanford University · 10 papers
- University of Oxford · 6 papers
- Microsoft · 6 papers
- Massachusetts Institute of Technology · 5 papers
- University of Cambridge · 5 papers
- Harvard University · 5 papers
- University of California, Berkeley · 5 papers
- McGill University · 4 papers
- Google · 4 papers
- University of Washington · 4 papers
Use it
Download BibTeX of 34 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Human-centered trust and agency (161 papers)
- Social-science view of AI evaluation (159 papers)
- Critiques of LLM claims and judging (104 papers)
- Benchmarking and reproducibility practice (104 papers)
- AI for health and public decisions (55 papers)
Elsewhere in the topic
- Teacher needs and AI curricula (45 papers)
- Classroom tools and student perceptions (40 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).