Research map / Large language models / AI and society
Fostering the Ecosystem of AI for Social Impact Requires Expanding and Strengthening Evaluation Standards
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
A position paper arguing that AI for social impact should recognize non-deployment and non-method contributions and adopt more rigorous, RCT- and event-study-style evaluation standards for deployments.
Problem
Review criteria and incentives in AI/ML for social impact favor projects that combine deployment with novel methods, undervaluing applied, non-deployed, or simple-method contributions and under-rigorously evaluating deployments.
Approach
Argues for an expansive notion of social impact beyond deployment (non-method contributions, non-deployment methodological impact such as simple/maintainable methods and single-variable benchmarks) and for field-experiment-grade evaluation of deployments (preregistration, power analysis, outcome validity, heterogeneity, RCTs, and event studies with counterfactuals).
Key results
- No empirical experiments; the paper's contribution is a set of recommendations for authors, reviewers, and venues.
- Recommends that ML venues require authors to declare whether field experiments were preregistered.
- Recommends normalizing simple domain-driven single-variable baselines in AISI papers.
Limitations
- No empirical evaluation of the proposed recommendations.
- Argument is structural and based on the authors' analysis and informal conversations with practitioners.
Experimental setup
Builds on: randomized controlled trials interrupted time series differences-in-differences synthetic control
Closest papers by meaning
- Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge (ICML 2025 · similarity 0.86)
- No Free Delivery Service: Epistemic limits of passive data collection in complex social systems (NeurIPS 2024 · similarity 0.83)
- Evaluating Index-based Treatment Allocation in Underresourced Communities (AAAI 2025 · similarity 0.83)
- MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research (NeurIPS 2025 · similarity 0.83)
- Scalable, Sustainable, Generalizable, and Responsible AI for Public Sector (AAAI 2025 · similarity 0.83)
- EVALUATION CARDS: An Interpretive Layer for AI Evaluation Reporting (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.