Research map / Multi-agent systems and games / Human-AI and goal-conditioned
AI Agents Push Humans Out of the Loop
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
A position paper arguing that current AI agent design and deployment undermine human oversight and degrade overseers' cognition, and proposing developer affordances and deployer organizational protocols as cognitive scaffolding.
Problem
Human-in-the-loop oversight of AI agents is assumed to be effective, but agent design impedes it and extended use of automation degrades the cognitive skills (situational awareness, critical judgement, domain skill) that oversight requires.
Approach
Frames the 'irony of automation' for agentic AI: oversight degrades the overseer. Proposes treating human cognitive requirements as first-class design constraints, via a goals-by-solutions inventory of developer runtime affordances (strategic friction, decision design, behavioral monitoring) and deployer protocols (trainings, workload scheduling, role design).
Key results
- Argues that current AI agent development and deployment do not support effective human oversight and contribute to its degradation.
- Proposes a two-pronged cognitive scaffolding approach: developer runtime affordances and deployer organizational protocols.
- Organizes solutions across six oversight goals and six solution categories (strategic friction, decision design, behavioral monitoring, trainings and exercises, workload and scheduling, role design).
Surprising findings
- Users may disprefer systems that reduce overreliance, creating a tension between user satisfaction and oversight preservation.
- Preference-based training could reward conditions of ineffective oversight if approval signals are not audited against independent oversight measures.
Limitations
- No empirical evaluation; the proposed solutions are not tested.
- Inventory is explicitly a structural map, not a definitive taxonomy.
- Relies on self-reports and emerging evidence (some EEG findings) that may be preliminary.
Experimental setup
Builds on: irony of automation (Bainbridge) dual-process theory EU AI Act human oversight requirements
Closest papers by meaning
- Position: Intent-aligned AI Systems Must Optimize for Agency Preservation (ICML 2024 · similarity 0.87)
- Towards a Theory of AI Personhood (AAAI 2025 · similarity 0.86)
- Calibrating Conservatism for Scalable Oversight (ICML 2026 · similarity 0.85)
- Towards a Science of AI Agent Reliability (ICML 2026 · similarity 0.85)
- JobBench: Aligning Agent Work With Human Will (NeurIPS 2026 · similarity 0.85)
- Position: Towards Bidirectional Human-AI Alignment (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.