Research map / Large language models / AI and society

Position: AI's growing due process problem

ICML 2025 · Large language models › AI and society · approach: Human-centered trust and agency

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

A position paper arguing that current AI systems, whose decisions cannot be explained or contested, cannot guarantee US due process and should not be used for decisions subject to it, grounded in two court cases (KW v. Armstrong, State v. Loomis).

Problem

AI-driven decision-making increasingly replaces human decision-makers in areas subject to US due process (notice, reasons, and the right to contest), but opaque models and trade secret protections prevent affected people and courts from understanding or challenging decisions.

Approach

Argues that the incompatibility between due process and modern AI is not just a transparency problem fixable by regulation, since even creators cannot causally explain large model decisions; combined with trade secret protections, this makes AI unfit for due-process-governed decisions, and calls on technical researchers to inform the legal process.

Key results

• Argues that AI systems whose behavior cannot be explained even by their creators are fundamentally incompatible with due process, not merely a transparency problem.
• Case study of KW v. Armstrong (Idaho): IDHW's AI-driven budget tool, built on 733 data points, provided notice that made it very difficult to determine why budgets were reduced; the court rejected trade secret shielding and the plaintiffs won.
• Case study of State v. Loomis (Wisconsin 2016): the court upheld COMPAS risk scores under trade secret protection, with cautionary advisements provided to judges, which the paper critiques.

Surprising findings

• The paper notes courts rejected trade secret shielding in KW v. Armstrong but upheld it for COMPAS in State v. Loomis.
• The author notes there is no comprehensive study of whether judges are resistant to cognitive bias when shown racially biased AI output with a warning label.

Limitations

• No new empirical experiments; argument rests on legal analysis and selected case studies.
• Relies on a small number of court cases and news reports, with some details of the cases unclear (e.g., the Idaho tool's implementation).

Experimental setup

Builds on: Citron (2007) technological due process Rudin (2019) interpretable models argument Goldberg v. Kelly (1970)

Closest papers by meaning

• Explainability matters: The effect of liability rules on the healthcare sector (NeurIPS 2026 · similarity 0.81)
• Position: If Innovation in AI systematically Violates Fundamental Rights, Is It Innovation at All? (NeurIPS 2025 · similarity 0.79)
• Regulating AI: Applying Insights from Behavioural Economics and Psychology to the Application of Article 5 of the EU AI Act (AAAI 2024 · similarity 0.79)
• AI Governance and Lessons Learned as an AI Policy Advisor in the United States Senate (AAAI 2025 · similarity 0.79)
• Position: A Safe Harbor for AI Evaluation and Red Teaming (ICML 2024 · similarity 0.78)
• Position: The Right to AI (ICML 2025 · similarity 0.78)