Research map / Large language models / Agents, code and math
Detecting Bugs with Substantial Monetary Consequences by LLM and Rule-based Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ABAuditor, a hybrid system where GPT-3.5 annotates financial meanings of entry-function parameters and globals, rule-based reasoning propagates meanings and flags accounting bugs, and reasoning-trace self-reflection curbs hallucinations.
Problem
Detecting accounting bugs (incorrect implementations of domain-specific financial models) in Solidity smart contracts, which existing tools and naive LLM prompting handle poorly due to context limits, hallucinations, and lack of generalization.
Approach
Split accounting-bug detection into LLM-based financial-meaning annotation of entry variables and deterministic rule-based propagation through the code (via Slither IR and SSA), with a trace-based LLM self-reflection loop to correct mislabeled annotations and reduce false positives.
Key results
- Recall of 90.5% (19 of 21 in-scope accounting bugs) on 23 projects, with 10 false positives versus 7 for the manual-annotation baseline.
- Financial-meaning annotation accuracy of 75.6% (62/82) against human annotations; fine-tuned GPT-4o mini with few-shot reaches 86.6% (71/82).
- Reasoning-trace feedback reduces false positives from 22 to 10 (54.5% reduction) without losing true positives; on 8 new projects it finds 6 of 7 in-scope bugs, including 2 zero-day.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LLM annotations with rule-based reasoning and reasoning-trace hallucination remedy instead of human annotations | ABAuditor · 23 smart contract projects | recall: 90.5% vs 90.5% | No clear effect |
| Use LLM annotations with rule-based reasoning and reasoning-trace hallucination remedy instead of human annotations | ABAuditor · 23 smart contract projects | false positives: 7 -> 10 | Mixed |
| Add reasoning-trace hallucination remedy (ablation) | ABAuditor · 23 smart contract projects | false positives: 22 -> 10; 54.5% reduction | Helped |
| Remove few-shot examples (ablation) | ABAuditor with GPT-3.5 · Human-labeled financial annotations | correct annotations: 62/82 -> 32/82 | Hurt |
| Add fine-tuning without few-shot examples (ablation) | ABAuditor with GPT-3.5 · Human-labeled financial annotations | correct annotations: 62/82 -> 39/82 | Hurt |
| Add fine-tuning while retaining few-shot examples (ablation) | ABAuditor with GPT-3.5 · Human-labeled financial annotations | correct annotations: 62/82 -> 64/82; 75.6% to 78% | Helped |
| Replace GPT-3.5 with fine-tuned GPT-4o mini and provide few-shot examples (ablation) | ABAuditor · Human-labeled financial annotations | correct annotations: 64/82 -> 71/82; 78% -> 86.6% | Helped |
| Replace GPT-3.5 with CodeLlama-13B (ablation) | ABAuditor · Human-labeled financial annotations | annotation accuracy: 41.9% vs 75.6% | Hurt |
| Rewrite variable names while preserving program semantics (ablation) | ABAuditor · Human-labeled financial annotations | annotation accuracy: 75.6% -> 31.7% | Hurt |
| Rewrite variable names while preserving program semantics (ablation) | ABAuditor · 23 smart contract projects | true positives: 19 -> 14 | Hurt |
| Rewrite variable names while preserving program semantics (ablation) | ABAuditor · 23 smart contract projects | false positives: 9 -> 21 | Hurt |
| Apply ABAuditor to recent projects with LLM annotation | ABAuditor · 8 new smart contract projects | in-scope bugs detected: 6 of the 7 | Helped |
Surprising findings
- Renaming variables to generic names (v1, v2) dropped annotation accuracy from 75.6% to 31.7%, showing heavy reliance on semantic naming.
- CodeLlama-13B achieved only 41.9% annotation accuracy, indicating the underlying LLM's capability matters substantially.
Limitations
- Only 6 financial meanings modeled, so contracts with other financial concepts may be missed or produce false positives.
- Rule-based reasoning is path-insensitive and cannot handle variables with multiple meanings from conditionals.
- Cannot detect pure-math accounting errors and only supports Solidity.
Experimental setup
Datasets: Zhang et al. 2023 smart contract bug dataset (513 bugs, 113 projects; 72 accounting bugs) 8 recent Code4rena projects (2024)
Benchmarks: Zhang et al. 2023 accounting bug dataset (23 projects) 8 new Code4rena smart contract projects
Metrics: recall false positives annotation accuracy tokens runtime API requests
Baselines: manual annotation with rule-based reasoning GPTScan ItyFuzz GPT-3.5 no few-shot CodeLlama-13B-instruct
Builds on: Slither GPTScan Zhang 2024 type system for accounting bugs chain-of-thought
Closest papers by meaning
- SCALM: Detecting Bad Practices in Smart Contracts Through LLMs (AAAI 2025 · similarity 0.88)
- MTVHunter: Smart Contracts Vulnerability Detection Based on Multi-Teacher Knowledge Translation (AAAI 2025 · similarity 0.86)
- SCDBench: A Benchmark for LLM-Based Smart Contract Decompilers (NeurIPS 2026 · similarity 0.84)
- IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities (ICLR 2025 · similarity 0.83)
- SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks (NeurIPS 2025 · similarity 0.83)
- Identifying the Risks of LM Agents with an LM-Emulated Sandbox (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.