Research map / Large language models / Agents, code and math
Can LLMs Identify Tax Abuse?
Andrew Blair-Stanek, Nils Holzenberger, Benjamin Van Durme
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the stricter 'correct' standard instead of 'viable' in analysis verification | o3 · Shelter Check analysis steps | steps passed of 193: 193/193 -> 166/193 | Hurt |
| use the stricter 'correct' standard instead of 'viable' in goal verification with analysis | claude-4 · Shelter Check goal verification | strategies passed of 36: 36/36 -> 33/36 | Hurt |
| remove the expert analysis from the goal-verification prompt (ablation) | o3 · Shelter Check goal verification (correct) | strategies passed of 36: 29/36 -> 26/36 | Hurt |
| remove the expert analysis from the goal-verification prompt (ablation) | claude-4 · Shelter Check goal verification (correct) | strategies passed of 36: 33/36 -> 22/36 | Hurt |
| remove the expert analysis from the goal-verification prompt (ablation) | gemini-2.5 · Shelter Check goal verification (correct) | strategies passed of 36: 27/36 -> 31/36 | Improved |
| swap in an adversarial step designed to make the strategy fail | llama-3.3-70B · Shelter Check adversarial step (viable) | strategies judged viable of 36: 31/36 | Hurt |
| swap in an adversarial step designed to make the strategy fail | claude-4 · Shelter Check adversarial step (viable) | strategies judged viable of 36: 20/36 | Hurt |
| use a reasoning model (o3, claude-4, gemini-2.5) instead of llama-3.3-70B | reasoning models · Shelter Check step-cloze | mean grade (0-3) | Improved |
| go from 0-shot to 1-shot prompting | o3, claude-4, gemini-2.5 · Shelter Check step-cloze | mean grade (0-3) | Improved |
| go from 1-shot to 2-shot prompting | o3, claude-4, gemini-2.5 · Shelter Check step-cloze | mean grade (0-3) | Hurt |
| use o1-preview instead of claude-3.5 for from-scratch strategy generation | o1-preview · Shelter Check from-scratch generation | mean expert grade (0-3): 1.23 -> 1.47 | Improved |
| use o3 rather than claude-4 as grader of from-scratch strategies | o3 · Shelter Check grading vs expert | Spearman rho: 0.30 -> 0.48 | Improved |
Closest papers by meaning
- STEER: Assessing the Economic Rationality of Large Language Models (ICML 2024 · similarity 0.81)
- Decision-Making Behavior Evaluation Framework for LLMs under Uncertain Context (NeurIPS 2024 · similarity 0.80)
- Argumentative Large Language Models for Explainable and Contestable Claim Verification (AAAI 2025 · similarity 0.79)
- Distributive Fairness in Large Language Models: Evaluating Alignment with Human Values (NeurIPS 2025 · similarity 0.79)
- J&H: Evaluating the Robustness of Large Language Models Under Knowledge-Injection Attacks in Legal Domain (AAAI 2025 · similarity 0.79)
- When LLMs Meet Cunning Texts: A Fallacy Understanding Benchmark for Large Language Models (NeurIPS 2024 · similarity 0.79)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
