Research map / Large language models / AI and society
To Err Is AI: A Case Study Informing LLM Flaw Reporting Practices
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reports lessons learned from a vendor panel running the open-ended DEF CON 2024 Generative Red Team 2 bug bounty against the OLMo language model, distilling best practices for LLM flaw reporting.
Problem
Flaw reporting for LLMs lacks established infrastructure, disclosure culture, and adjudication practices, so safety failures relative to model documentation are hard to report and evaluate by outside participants.
Approach
Operationalizes a community bug bounty for LLM flaws (rather than vulnerabilities) by having a vendor panel adjudicate hacker reports against model documentation, and distills six operational challenges and lessons from it, such as systematic-vs-individual failure criteria, two-tier bounties, and documentation changes.
Key results
- 495 participants produced 200 flaw reports during GRT2 at DEF CON 2024, with $7,400 of the $10,000 prize pool paid out.
- The most common rejection was proof-of-concept single-prompt jailbreaks rather than systematic flaws; a two-tier bounty ($50 single-prompt, $500 systematic) was introduced.
- The OLMo model card was updated to explicitly disclaim legal advice use cases after participants showed the model giving confabulated legal advice.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use WildGuard-triggered reprompting instead of inserting a guaranteed refusal | OLMo with WildGuard · Jailbreak attacks | susceptibility to jailbreak attacks | Hurt |
| Disclaim legal advice as an intended use case in the model card | OLMo documentation · Legal advice | likelihood of replicating legal-advice harms | Helped |
| Introduce a two-tier bounty system for single-prompt and systematic demonstrations | GRT2 adjudication process · Participant engagement | participant persistence | Helped |
| Introduce a two-tier bounty system that awards small bounties for interesting single-prompt submissions | GRT2 adjudication process · Jailbreak submissions | mass-produced single-prompt jailbreak submissions | Hurt |
| Require jailbreak attacks to demonstrate general failure for the larger bounty | GRT2 adjudication process · Jailbreak submissions | systematic failure demonstrations | Helped |
Surprising findings
- A tiered bounty for single-prompt jailbreaks opened the door to mass-produced individual instances, requiring the jailbreak category to be restricted to systematic failures on day one.
- Automated scoring mechanisms that did not rely on human annotation proved more confusing than useful for submitters.
Limitations
- Single event with a homogeneous hacker population, so adjudication generalizability to broader users is unclear.
- Lessons come from one vendor panel's perspective and an event with a three-month planning window.
Experimental setup
Metrics: number of flaw reports bounty payouts accepted submissions
Builds on: coordinated flaw disclosure framework AI Incident Database WildGuard
Closest papers by meaning
- Position: In-House Evaluation Is Not Enough. Towards Robust Third-Party Evaluation and Flaw Disclosure for General-Purpose AI (ICML 2025 · similarity 0.86)
- Position: Editing Large Language Models Poses Serious Safety Risks (ICML 2025 · similarity 0.84)
- AI Control: Improving Safety Despite Intentional Subversion (ICML 2024 · similarity 0.84)
- SafetyPrompts: A Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety (AAAI 2025 · similarity 0.84)
- On the Consideration of AI Openness: Can Good Intent Be Abused? (AAAI 2025 · similarity 0.82)
- SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.