Research map / Large language models / Agents, code and math
Decompile-Bench: Million-Scale Binary-Source Function Pairs for Real-World Binary Decompilation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Decompile-Bench, a two-million-pair binary-source function dataset from permissively licensed C/C++ GitHub projects, plus a leakage-resistant evaluation suite, and shows fine-tuning LLM4Decompile on 10% of it improves re-executability by about 20%.
Problem
Public large-scale real-world binary-source function pairs are lacking for training and evaluating LLM-based decompilers, since DWARF-based mapping is incomplete under optimization and inlining and prior benchmarks are contest-style, synthetic, or fragment-level.
Approach
A Compile-Trace-Filter pipeline that forks Clang to force -O0-3 and -g builds, maps each binary function to its complete source function by intersecting DWARF line segments with Tree-sitter-parsed function bodies, and then applies project-scope, in-binary and cross-binary (MinHash-LSH) deduplication filters.
Key results
- Fine-tuning LLM4Decompile-End-1.3b on 10% of Decompile-Bench raised average re-executability on HumanEval from 16.22% to 20.89% and on MBPP from 20.54% to 24.93%.
- LLM4Decompile-DCBench-6.7b reaches 39.48% average re-executability on HumanEval, versus 46.79% for Claude-Sonnet-4-reasoning and 18.22% for IDA.
- Training on the unfiltered raw data lowered re-executability relative to the base model (15.86% vs 16.22% on HumanEval), while the filtered Decompile-Bench data improved it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune LLM4Decompile-End-1.3b on Decompile-Bench | LLM4Decompile-End-1.3b · HumanEval | Re-Executability (AVG): 16.22 -> 20.89 | Helped |
| Fine-tune LLM4Decompile-End-1.3b on Decompile-Bench | LLM4Decompile-End-1.3b · MBPP | Re-Executability (AVG): 20.54 -> 24.93 | Helped |
| Train on ExeBench (ablation) | LLM4Decompile-End · HumanEval and MBPP | Re-Executability (AVG): 16.22 -> 16.78; 20.54 -> 20.20 | No clear effect |
| Train on unfiltered Decompile-Bench-raw (ablation) | LLM4Decompile-End · HumanEval and MBPP | Re-Executability (AVG): 16.22 -> 15.86; 20.54 -> 18.17 | Hurt |
| Train on filtered Decompile-Bench (ablation) | LLM4Decompile-End · HumanEval and MBPP | Re-Executability (AVG): 16.22 -> 20.89; 20.54 -> 24.93 | Helped |
| Train on ExeBench (ablation) | LLM4Decompile-End · HumanEval, MBPP, and GitHub2025 | R2I (AVG): 54.57 -> 54.40; 55.75 -> 56.11; 48.71 -> 47.97 | Mixed |
| Train on unfiltered Decompile-Bench-raw (ablation) | LLM4Decompile-End · HumanEval, MBPP, and GitHub2025 | R2I (AVG): 54.57 -> 55.33; 55.75 -> 56.83; 48.71 -> 66.10 | Helped |
| Train on filtered Decompile-Bench (ablation) | LLM4Decompile-End · HumanEval, MBPP, and GitHub2025 | R2I (AVG): 54.57 -> 60.36; 55.75 -> 61.81; 48.71 -> 70.89 | Helped |
| Train on ExeBench (ablation) | LLM4Decompile-End · HumanEval, MBPP, and GitHub2025 | Edit Similarity (AVG): 38.34 -> 37.10; 40.41 -> 40.69; 21.57 -> 21.24 | Mixed |
| Train on unfiltered Decompile-Bench-raw (ablation) | LLM4Decompile-End · HumanEval, MBPP, and GitHub2025 | Edit Similarity (AVG): 38.34 -> 39.82; 40.41 -> 42.08; 21.57 -> 29.01 | Helped |
| Train on filtered Decompile-Bench (ablation) | LLM4Decompile-End · HumanEval, MBPP, and GitHub2025 | Edit Similarity (AVG): 38.34 -> 46.22; 40.41 -> 49.28; 21.57 -> 29.51 | Helped |
| Train a binary-source embedding model on 10% of Decompile-Bench | Binary-source embedding model · GitHub2025 | Recall@1: 27.0 | Helped |
Surprising findings
- Training on unfiltered Decompile-Bench-raw data reduced re-executability versus the base model, while readability metrics rose, indicating a correctness-readability trade-off.
- ExeBench's executable subset is substantially less complex than general GitHub functions in cyclomatic complexity and Halstead difficulty.
Limitations
- Only 10% of the dataset was used, and only a 1.3B model was trained on it due to compute limits.
- Re-executability is evaluated only on small HumanEval/MBPP-style functions, not on real-world project tests; the authors note difficulty building such tests.
- About 25% of sampled pairs contain compiler-induced scope mismatches from function inlining.
Experimental setup
Datasets: Decompile-Bench ExeBench HumanEval MBPP ProRec
Benchmarks: Decompile-Bench-Eval (HumanEval-C/C++, MBPP-C/C++, GitHub2025) ProRec
Metrics: re-executability R2I edit similarity CodeSage embedding similarity CodeBLEU GPT-judge score recall@1
Baselines: IDA GPT-4.1-mini Claude-Sonnet-4-reasoning Idioms-1.3b LLM4Decompile-End-1.3b Ghidra
Builds on: LLM4Decompile Idioms ExeBench Tree-sitter LLaMA-Factory
Closest papers by meaning
- ASAP: Assembly-Source Aligned Pseudocode Refinement For Binary Decompilation (NeurIPS 2026 · similarity 0.86)
- SCDBench: A Benchmark for LLM-Based Smart Contract Decompilers (NeurIPS 2026 · similarity 0.85)
- Yggdrasil: Bridging Dynamic Speculation and Static Runtime for Latency-Optimal Tree-Based LLM Decoding (NeurIPS 2025 · similarity 0.84)
- Source Code Foundation Models are Transferable Binary Analysis Knowledge Bases (NeurIPS 2024 · similarity 0.84)
- Fast-dLLM++: Fr\'{e}chet Profile Decoding for Faster Diffusion LLM Inference (NeurIPS 2026 · similarity 0.84)
- The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.