Research map / Large language models / Language, safety and interpretability

SCDBench: A Benchmark for LLM-Based Smart Contract Decompilers

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SCDBench, a 600-contract benchmark with staged metrics (format, compilability, ABI recovery, differential semantic replay) for evaluating LLM-based smart contract decompilers, showing frontier LLMs compile often but rarely decompile semantically perfectly.

Problem

Evaluating smart contract decompilers is hard: existing studies use narrow datasets, inconsistent metrics, and weak semantic checks, while LLMs now produce plausible-looking Solidity that may not preserve original semantics.

Approach

A benchmark dataset of 600 verified Ethereum contracts with bytecode, ground-truth Solidity, and replayable function-level semantic test cases, paired with a cumulative four-stage evaluation pipeline (format, compilability, ABI F1, semantic replay) plus a single-step same-model compilation-repair variant.

Key results

• Opus 4.7 with repair achieves the best total semantic consistency at 29.4% of 14,553 public functions (4,284), but only 42/600 contracts are perfectly decompiled.
• GPT-5.3-Codex with repair reaches 90.3% compilability (542/600) and ABI micro-F1 0.896, up from 70.2% compilability unrepaired.
• Same-model compilation repair improves every model, e.g., GLM-5 compilability rises from 36.8% to 67.2%.

Surprising findings

• The model with the best compilability and ABI recovery (GPT-5.3-Codex) is not the one with the best semantic consistency overall (Opus 4.7 with repair).
• Extended reasoning did not improve format compliance: GLM-5 (instruct) had higher format completeness than GLM-5 with high thinking.

Limitations

• Semantic replay covers only tested inputs and reaches about 77.8% code coverage on average, so it is not a proof of semantic equivalence.
• Semantic replay requires an Ethereum archive node, adding infrastructure cost.
• Traditional decompilers are only evaluated on ABI recovery because they do not produce compilable Solidity.

Experimental setup

Datasets: verified Ethereum Solidity contracts (crawled, 772,736 unique bytecode-source pairs)

Benchmarks: SCDBench

Metrics: format completeness compilability ABI precision/recall/F1 semantic consistency rate perfect contract rate

Baselines: Claude Opus 4.7 GPT-5.3-Codex GLM-5 GLM-5 (instruct) Gigahorse Heimdall-rs

Builds on: Gigahorse Elipmoc Shrnkr DiSCo Heimdall-rs

Closest papers by meaning

• SCALM: Detecting Bad Practices in Smart Contracts Through LLMs (AAAI 2025 · similarity 0.87)
• BaxBench: Can LLMs Generate Correct and Secure Backends? (ICML 2025 · similarity 0.86)
• SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks (NeurIPS 2025 · similarity 0.85)
• MTVHunter: Smart Contracts Vulnerability Detection Based on Multi-Teacher Knowledge Translation (AAAI 2025 · similarity 0.85)
• Decompile-Bench: Million-Scale Binary-Source Function Pairs for Real-World Binary Decompilation (NeurIPS 2025 · similarity 0.85)
• Beyond Accuracy: Evaluating Self-Consistency of Code Large Language Models with IdentityChain (ICLR 2024 · similarity 0.84)