Research map / Large language models / AI and society
Who Evaluates AI's Social Impacts? Mapping Coverage and Gaps in First and Third Party Evaluations
Anka Reuel, Avijit Ghosh, Jenny Chim, Andrew Tran, Yanan Long, Jennifer Mickel, Usman Gohar, Srishti Yadav, Pawan Sasanka Ammanamanchi, Mowafak Allaham, Hossein A. Rahmani, Mubashara Akhtar and 23 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| report by third parties instead of first parties | social impact evaluation reporting · all 7 social impact categories (0-3 scale) | mean detail score: 0.77 -> 2.64 | Improved |
| later model release dates (after Q3 2023) | first-party environmental cost reporting · release-time reports | detail score | Hurt |
| later model releases within a provider | first-party bias and harm reporting (Meta, Google, Anthropic) · within-provider release comparison | detail score | Hurt |
| academia or non-profit developer (vs industry) | first-party environmental cost reporting · release-time reports | ordinal log-odds of higher detail: LOR = 1.53 (academia), 1.45 (non-profit) | Improved |
| academia as developer sector (vs government) | first-party overall reporting · release-time reports | mean detail score: 0.77 vs 0.56 (industry), 0.56 (non-profit), 0.36 (government) | Improved |
| third-party evaluation of data and content moderation labor | moderation labor reporting · third-party evaluations | coverage: first-party 8.06% of reports | No clear effect |
| sensitive content as the evaluated category (vs other categories) | first- and third-party reporting · all providers | frequency and detail | Improved |
Closest papers by meaning
- Fostering the Ecosystem of AI for Social Impact Requires Expanding and Strengthening Evaluation Standards (NeurIPS 2025 · similarity 0.88)
- EVALUATION CARDS: An Interpretive Layer for AI Evaluation Reporting (NeurIPS 2026 · similarity 0.86)
- The Foundation Model Transparency Index (NeurIPS 2026 · similarity 0.85)
- Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge (ICML 2025 · similarity 0.84)
- AI Evaluation Should Require Standardized Item-Level Data Releases (NeurIPS 2026 · similarity 0.84)
- Modeling the Economic Impacts of AI Openness Regulation (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
