Research map / Large language models / AI and society
Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper arguing that rigor in AI should extend beyond methodological rigor to six facets (epistemic, normative, conceptual, methodological, reporting, interpretative), drawing on responsible AI literature.
Problem
Rigor in AI research and practice is largely understood as methodological rigor, which leaves unaddressed background assumptions, value choices, construct clarity, reporting, and claim interpretation, contributing to overblown capability claims and harms.
Approach
Proposes a six-facet framework of rigor, each with an object of concern and evaluative questions, and argues responsible AI concerns are within the scope of rigorous AI work for all researchers and practitioners.
Key results
- Identifies six facets of rigor (epistemic, normative, conceptual, methodological, reporting, interpretative) with objects of concern and evaluative questions in a summary table.
- Argues that the narrow methodological conception of rigor contributes to concerns raised by the responsible AI community, such as overblown capability claims.
- Illustrates facets with examples such as hallucination as a contested construct and mathematical reasoning benchmark claims.
Limitations
- Does not prescribe rigid criteria; offers scaffolding rather than an operational standard
- Acknowledges rigor is not a panacea and the facets are not claimed to be exhaustive
- Facets are tangled and difficult to separate cleanly
Experimental setup
Builds on: measurement validity (consequential validity) positionality statements pre-registration datasheets for datasets model cards
Closest papers by meaning
- Embracing Contradiction: Theoretical Inconsistency Will Not Impede the Road of Building Responsible AI Systems (NeurIPS 2025 · similarity 0.88)
- Towards a Science of AI Agent Reliability (ICML 2026 · similarity 0.86)
- Towards a Theory of AI Personhood (AAAI 2025 · similarity 0.85)
- ACED-Bench: Evaluating How LLM Agents Acquire and Act on Causal Evidence (NeurIPS 2026 · similarity 0.85)
- Regulating AI: Applying Insights from Behavioural Economics and Psychology to the Application of Article 5 of the EU AI Act (AAAI 2024 · similarity 0.85)
- MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.