Research map / Large language models / Agents, code and math
FindTheFlaws: Annotated Errors for Detecting Flawed Reasoning and Scalable Oversight Research
Gabriel Recchia, Chatrik Singh Mangat, Issac Li, Gayatri Krishnakumar
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use o3-mini instead of Claude 3.5 Sonnet as the judge | match task · Adversarial MedQA | F1: 0.611 -> 0.780 | Improved |
| use Claude 3.7 Sonnet instead of Claude 3.5 Sonnet as the judge | match task · GPQA Diamond Plus | F1: 0.531 -> 0.700 | Improved |
| use o3-mini instead of GPT-4o on standard Python650 | match task · Std. Python650 | F1: 0.741 -> 0.971 | Improved |
| compare model to human expert baseline on Adversarial MedQA | o3-mini · Adversarial MedQA | F1: 0.548 -> 0.780 | Improved |
| compare Claude 3.7 Sonnet to human baseline on CELS Lojban | Claude 3.7 Sonnet · CELS Lojban match-all | F1: 0.921 -> 0.765 | Hurt |
| compare o3-mini to human baseline on CELS Law | o3-mini · CELS Law match-all | F1: 0.907 -> 0.777 | Hurt |
| compare o3-mini to human validator on CELS Surgery | o3-mini · CELS Surgery match-all | F1: 0.940 -> 0.936 | No clear effect |
| move from standard to Meta Python650 for o3-mini | o3-mini · Python650 | F1: 0.971 -> 0.862 | Hurt |
| use error-grading instead of match task for the human baseline | human expert · Adversarial MedQA | score: 0.548 -> 0.344 | Hurt |
| use Alt Meta Python650 (correct answer, flawed reasoning) instead of Meta Python650 | human baseline · error grading | accuracy: 0.596 -> 0.264 | Hurt |
Closest papers by meaning
- On scalable oversight with weak LLMs judging strong LLMs (NeurIPS 2024 · similarity 0.88)
- Great Models Think Alike and this Undermines AI Oversight (ICML 2025 · similarity 0.87)
- Scaling Laws For Scalable Oversight (NeurIPS 2025 · similarity 0.86)
- RFEval: Benchmarking Reasoning Faithfulness under Counterfactual Reasoning Intervention in Large Reasoning Models (ICLR 2026 · similarity 0.86)
- An Enigma of Artificial Reason: Investigating the Production-Evaluation Gap in Large Reasoning Models (NeurIPS 2026 · similarity 0.86)
- Decoding the Critique Mechanism in Large Reasoning Models (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
