Research map / Label-efficient and robust learning / Recognition and long-tail
Assemblage: Automatic Binary Dataset Construction for Machine Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents Assemblage, a distributed AWS-based system that crawls GitHub and vcpkg, builds Windows PE (and some Linux ELF) binaries with many compiler configurations, and releases a dataset plus reproducible "recipes", then benchmarks learning models on it.
Problem
Benign binary corpora for training machine learning binary analysis models are scarce and mostly Linux; Windows PE corpora with source-binary mappings and licensing-compliant distribution are lacking.
Approach
An extensible distributed build system that crawls open-source repositories, compiles them under many toolchain/optimization configurations, and records function-level ground truth (PDBs, RVAs, source lines) in a SQLite database, distributing only permissively licensed binaries and publishing recipes for the rest.
Key results
- Assemblage produced 890k Windows PE and 428k Linux ELF binaries across 29 configurations, with 1.5M+ total binaries in the comparison table.
- PassTell reaches only about 58% accuracy when trained and tested on Assemblage Windows PE binaries.
- jTrans achieves MRR 0.52 on Windows versus 0.83 on Linux after fine-tuning, and the GNN's AUC drops to 0.50-0.72 for cross-platform tasks such as arch and opt.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate PassTell on Windows PE binaries rather than relying on Linux training data | PassTell · Compiler provenance, Windows PE | accuracy: ≈58% | Hurt |
| Train on Windows and test across the Windows/Linux domain boundary | GNN · Binary function similarity, multiple tasks | AUC: arch: 0.97 -> 0.50; bit: 0.99 -> 0.70; comp: 0.80 -> 0.63; opt: 0.88 -> 0.72; ver: 0.98 -> 0.82; XA: 0.86 -> 0.48; XC: 0 | Hurt |
| Train on Linux and test on Windows | GNN · Binary function similarity, opt, ver, XM | AUC: opt: 0.88 -> 0.62; ver: 0.98 -> 0.64; XM: 0.87 -> 0.61 | Hurt |
| Fine-tune and evaluate jTrans on Windows rather than Linux | jTrans · Function similarity | MRR: 0.83 -> 0.52 | Hurt |
| Use PDB debug information with IDA | IDA · Function boundary identification, WinPE | F-1: 0.47 -> 0.79 | Helped |
| Use PDB debug information with IDA | IDA · Function boundary identification, vcpkg | F-1: 0.86 -> 0.86 | No clear effect |
| Allow singleton functions to remain in the dataset and participate in negative pairs | GNN binary function similarity pipeline · Binary function similarity | performance | Helped |
| Increase Windows fine-tuning from 1 to 20 epochs | jTrans · Function similarity, Windows | MRR: 0.17 -> 0.52 | Helped |
| Increase Windows fine-tuning from 20 to 50 epochs | jTrans · Function similarity, Windows | MRR: 0.52 -> 0.52 | No clear effect |
| Increase Linux fine-tuning from 1 to 50 epochs | jTrans · Function similarity, Linux | MRR: 0.83 -> 0.83 | No clear effect |
| Increase BinaryCorp-26M fine-tuning from 1 to 50 epochs | jTrans · Function similarity, BinaryCorp-26M | MRR: 0.82 -> 0.98 | Helped |
Surprising findings
- Number of functions in a binary is not correlated with binary size (Pearson 0.22), suggesting binary size is a weak proxy for function count.
- IDA Pro function boundary F1 on Windows PE is only 0.47 without PDBs, versus 0.79 with PDBs; XDA reaches 0.75 on WinPE.
Limitations
- Build success rate is low (about 30% on Windows and 7% on Linux), so many repositories do not yield binaries.
- No malware scanning is performed on crawled repositories.
- Only permissively licensed binaries are distributed publicly; unlicensed repositories require recipes.
Experimental setup
Datasets: Assemblage Windows-GitHub Assemblage vcpkg Assemblage Linux-GitHub BinaryCorp-26M vcpkg
Benchmarks: Assemblage Windows PE binaries Assemblage Linux ELF binaries vcpkg
Metrics: accuracy AUC MRR F1
Baselines: XDA IDA BinKit BinaryCorp-26M jTrans-zero
Builds on: jTrans PassTell XDA EMBER BinaryCorp
Closest papers by meaning
- Is Function Similarity Over-Engineered? Building a Benchmark (NeurIPS 2024 · similarity 0.86)
- Decompile-Bench: Million-Scale Binary-Source Function Pairs for Real-World Binary Decompilation (NeurIPS 2025 · similarity 0.83)
- A Large-Scale Multi-Source Dataset Linking Hacker Community Discourse to the CVE Vulnerability Lifecycle (NeurIPS 2026 · similarity 0.83)
- QUT-DV25: A Dataset for Dynamic Analysis of Next-Gen Software Supply Chain Attacks (NeurIPS 2025 · similarity 0.82)
- Source Code Foundation Models are Transferable Binary Analysis Knowledge Bases (NeurIPS 2024 · similarity 0.82)
- RustMizan: A Compilable, Contamination-Aware Benchmarking Framework for Rust Vulnerabilities (NeurIPS 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.