Research map / Large language models / AI and society
When Your AI Becomes a Target: AI Security Incidents and Best Practices
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compiles a database of 32 real-world AI security incidents (previous reports plus a new survey of 271 practitioners) and analyzes targets, attacker goals, correlates, causes, and mitigations, concluding that security and privacy best practices, especially access control, could have prevented many incidents.
Problem
Real-world AI security incidents are rarely reported in enough detail to understand attacker motives, targets, and how to avoid them, leaving a gap between academic AI security research and practice.
Approach
Combines previously reported AI incidents with freshly collected practitioner questionnaire reports into a small structured dataset, categorizing each incident by target (data, model, infrastructure), timing in the ML pipeline, and attacker goal (confidentiality, integrity, availability), and derives mitigations mainly from security and privacy best practices.
Key results
- Of 32 incidents, the most frequent target is data, followed by infrastructure and model; confidentiality is the most common attacker goal.
- Company size correlates with number of incidents (Spearman 0.29), while likelihood of victimhood correlates with incidents at 0.49 and time in production does not correlate.
- Only 3% of 271 new participants reported AI attacks, versus 15% in the prior Grosse et al. sample; 15 of 32 incidents judged avoidable with best practices.
Surprising findings
- Time the model has been in production does not correlate with the number of incidents.
- Many incidents stem from misconfiguration or cybersecurity lapses rather than AI-specific attacks.
Limitations
- Small sample of 32 incidents with limited detail; 25% unclear.
- Sample biased toward global north and Western respondents; English-only questionnaire.
- Low incident rate (3%) among recruited participants limits statistical power.
Experimental setup
Datasets: Grosse et al. 2023b incident reports Bieringer et al. 2022 incident new questionnaire responses from 271 participants
Metrics: incident counts by target, time, and goal spearman correlation
Builds on: Grosse et al. 2023b Bieringer et al. 2022 AI Incident Database
Closest papers by meaning
- Lessons for Editors of AI Incidents from the AI Incident Database (AAAI 2025 · similarity 0.86)
- Position: In-House Evaluation Is Not Enough. Towards Robust Third-Party Evaluation and Flaw Disclosure for General-Purpose AI (ICML 2025 · similarity 0.79)
- Evaluation and Incident Prevention in an Enterprise AI Assistant (AAAI 2025 · similarity 0.79)
- To Err Is AI: A Case Study Informing LLM Flaw Reporting Practices (AAAI 2025 · similarity 0.79)
- Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition (NeurIPS 2025 · similarity 0.79)
- AI Risk Profiles: A Standards Proposal for Pre-deployment AI Risk Disclosures (AAAI 2024 · similarity 0.78)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.