Research map / Large language models / Retrieval and knowledge
SCALM: Detecting Bad Practices in Smart Contracts Through LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SCALM, an LLM-based smart contract auditing framework combining RAG over a DAppSCAN-derived vector knowledge base with Step-Back Prompting to detect SWC-style bad practices and generate JSON audit reports.
Problem
Detecting bad coding practices in Ethereum smart contracts (SWC-categorized weaknesses that may not be direct security vulnerabilities), which existing pattern-matching tools handle poorly and with limited coverage.
Approach
Combine a vector database of code snippets with bad practices (retrieved via cosine similarity using embeddings) with Step-Back Prompting so the LLM abstracts high-level principles before reasoning about each code fragment, producing structured audit reports.
Key results
- On SmartBugs SWC-112, SCALM (GPT-4o) reaches 97.30% F1 versus 70.37% for the best baseline (Mythril).
- On SWC-104, SCALM achieves 98.27% F1 compared with 33.00% for Conkas.
- Ablation: removing RAG drops F1 on SWC-107 from 94.97% to 75.35% and on SWC-101 from 95.45% to 79.71%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SCALM instead of Oyente | Smart contract bad-practice detection · SWC-101 | F1 (%): 66.03 -> 95.45 | Helped |
| Use SCALM instead of Conkas | Smart contract bad-practice detection · SWC-104 | F1 (%): 33.00 -> 98.27 | Helped |
| Use SCALM instead of Conkas | Smart contract bad-practice detection · SWC-107 | F1 (%): 77.50 -> 94.97 | Helped |
| Use SCALM instead of Mythril | Smart contract bad-practice detection · SWC-112 | F1 (%): 70.37 -> 97.30 | Helped |
| Use SCALM instead of Conkas | Smart contract bad-practice detection · SWC-116 | F1 (%): 88.50 -> 92.67 | Helped |
| Add RAG (ablation) | SCALM with GPT-4o · SWC-101 | F1 (%): 79.71 -> 95.45 | Helped |
| Add RAG (ablation) | SCALM with GPT-4o · SWC-104 | F1 (%): 79.19 -> 98.27 | Helped |
| Add RAG (ablation) | SCALM with GPT-4o · SWC-107 | F1 (%): 75.35 -> 94.97 | Helped |
| Add RAG (ablation) | SCALM with GPT-4o · SWC-112 | F1 (%): 73.99 -> 97.30 | Helped |
| Add RAG (ablation) | SCALM with GPT-4o · SWC-116 | F1 (%): 83.13 -> 92.67 | Helped |
Surprising findings
- Gemini-1.5-Pro and Claude-3.5-Sonnet show uneven per-category detection, failing on categories GPT-4o handles, indicating model choice strongly affects results.
Limitations
- Struggles with state-dependent vulnerabilities requiring understanding of contract evolution over time.
- Limited understanding of blockchain mechanisms and complex interactions.
- Evaluation counts detections by SWC-ID or keyword matching, and comparison datasets are small per category.
Experimental setup
Datasets: DAppSCAN SmartBugs
Benchmarks: DAppSCAN SmartBugs
Metrics: accuracy recall F1
Baselines: Mythril Oyente Confuzzius Conkas
Builds on: retrieval-augmented generation step-back prompting Mythril
Closest papers by meaning
- Detecting Bugs with Substantial Monetary Consequences by LLM and Rule-based Reasoning (NeurIPS 2024 · similarity 0.88)
- SCDBench: A Benchmark for LLM-Based Smart Contract Decompilers (NeurIPS 2026 · similarity 0.87)
- MTVHunter: Smart Contracts Vulnerability Detection Based on Multi-Teacher Knowledge Translation (AAAI 2025 · similarity 0.84)
- EVMbench: Evaluating AI Agents on Smart Contract Security (ICML 2026 · similarity 0.83)
- BaxBench: Can LLMs Generate Correct and Secure Backends? (ICML 2025 · similarity 0.82)
- SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.