Research map / Large language models / AI and society
Social-science view of AI evaluation
Argue for rigorous, socially grounded measurement and broader evaluation of AI systems.
159 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in AI and society, keyword cluster “human · systems · argue”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
38 papers in 2023–24 and 109 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 16.8% to 21.3%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · On Human Intellect and Machine Failures: Troubleshooting Integrative Machine Learning Systems (AAAI 2017 · 63 citations)
- 2018 · State of the Art: Reproducibility in Artificial Intelligence (AAAI 2018 · 280 citations)
- 2019 · Bridging Machine Learning and Logical Reasoning by Abductive Learning (NeurIPS 2019 · 89 citations)
- 2020 · Asymptotically Unambitious Artificial General Intelligence (AAAI 2020 · 19 citations)
- 2021 · Aligning Artificial Neural Networks and Ontologies towards Explainable AI (AAAI 2021 · 44 citations)
- 2022 · Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization? (NeurIPS 2022 · 28 citations)
- 2023 · MoMusic: A Motion-Driven Human-AI Collaborative Music Composition and Performing System (AAAI 2023 · 18 citations)
- 2024 · The Alignment Problem from a Deep Learning Perspective (ICLR 2024 · 63 citations)
- 2025 · Why AI Is WEIRD and Shouldn't Be This Way: Towards AI for Everyone, with Everyone, by Everyone (AAAI 2025 · 22 citations)
- 2026 · Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents (ICLR 2026 · 22 citations)
Earliest papers here
- On Human Intellect and Machine Failures: Troubleshooting Integrative Machine Learning Systems (AAAI 2017 · 63 citations)
- State of the Art: Reproducibility in Artificial Intelligence (AAAI 2018 · 280 citations)
- Bridging Machine Learning and Logical Reasoning by Abductive Learning (NeurIPS 2019 · 89 citations)
What has been tried
77 papers here report 595 measured changes (a component added, removed or swapped, compared with a baseline): helped 58%, no clear effect 11%, hurt 24%, mixed 8%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Remove explicit date information from CNN news articles | CNN news articles · classification accuracy | 100% accuracy; one mistake and 3 ambiguous answers | Hurt | The Alignment Problem from a Deep Learning Perspective (ICLR 2024 · 63 citations) |
| Use deconfounded candidate responses instead of original human annotations | Object HalBench · response hallucination rate | 25.7 -> 10.1 | Helped | RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthiness (CVPR 2025 · 20 citations) |
| Replace realistic transition probabilities with unrealistic probabilities | Instrumentality analysis · instrumentality loss | Hurt | Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs (NeurIPS 2025 · 9 citations) | |
| Deduct points when the answer explicitly names a requested skill | Skill-Mix(k=2), GPT-4 grading · GPT-4 Ratio of Full Marks | .94 -> .91 | Hurt | SKILL-MIX: a Flexible and Expandable Family of Evaluations for AI Models (ICLR 2024 · 6 citations) |
| use evolutionary search rather than rewriting only the seed poem | Poetry · QD score | 99 (CI: 72 - 117) -> 130 (CI: 118 - 145) | Helped | Quality-Diversity through AI Feedback (ICLR 2024 · 6 citations) |
| Remove explanation request from the prompt | Number category · accuracy (%) | 49.5% to 17.6% | Hurt | Evaluating Large Vision-and-Language Models on Children's Mathematical Olympiads (NeurIPS 2024 · 6 citations) |
| Increase Arena data in the training mixture from 0% to 70% | ArenaHard · win-rate vs. Llama-3.1-8B-Instruct | 23.5% -> 49.9% | Helped | The Leaderboard Illusion (NeurIPS 2025 · 4 citations) |
| Add normalization against random-action success rates to the LP curriculum | Crafter · average task success rate and tasks learned | Helped | OMNI: Open-endedness via Models of human Notions of Interestingness (ICLR 2024 · 4 citations) | |
| use Claude-series instead of 4o-series as the research-agent backbone | Scientist-Bench evaluation subset · completeness | 50% -> 87.5% | Helped | AI-Researcher: Autonomous Scientific Innovation (NeurIPS 2025 · 3 citations) |
| add skill-aware attention | Maia Testset, Avg · top-1 accuracy | 51.75 -> 52.74 | Helped | Maia-2: A Unified Model for Human-AI Alignment in Chess (NeurIPS 2024 · 3 citations) |
| Fine-tune Mixtral8x7b on activation-description examples | Synthetic held-out data · Shape prediction | Helped | Neurons to Words: A Novel Method for Automated Neural Network Interpretability and Alignment (AAAI 2025 · 1 citation) | |
| Train on 50% of the dataset | Move-matching accuracy across skill levels · move-matching accuracy | -1.0% | Hurt | Human-Aligned Chess With a Bit of Search (ICLR 2025 · 1 citation) |
What reviewers will expect
Datasets and baselines named in the 83 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- MMLU · 6 of 83 papers
- GSM8K · 4 of 83 papers
- MATH · 2 of 83 papers
- MedQA · 2 of 83 papers
- WMDP · 2 of 83 papers
- GPQA · 2 of 83 papers
Baselines
- Claude 3.5 Sonnet · 5 of 83 papers
- GPT-4o · 4 of 83 papers
- Claude-3-Opus · 2 of 83 papers
- Gemini-1.5-Pro · 2 of 83 papers
- HELM · 2 of 83 papers
- MLP · 2 of 83 papers
- GPT-4o Mini · 2 of 83 papers
- o1 · 2 of 83 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 147 papers; from OpenAlex and the papers' affiliations).
- Stanford University · 16 papers
- Carnegie Mellon University · 14 papers
- Massachusetts Institute of Technology · 12 papers
- University of California, Berkeley · 10 papers
- Google · 8 papers
- Harvard University · 8 papers
- University of Illinois, Urbana Champaign · 8 papers
- Microsoft · 7 papers
- Cornell University · 7 papers
- Meta · 6 papers
Use it
Download BibTeX of 30 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Human-centered trust and agency (161 papers)
- Data commons and human baselines (159 papers)
- Critiques of LLM claims and judging (104 papers)
- Benchmarking and reproducibility practice (104 papers)
- AI for health and public decisions (55 papers)
Elsewhere in the topic
- Teacher needs and AI curricula (45 papers)
- Classroom tools and student perceptions (40 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).