Research map / Learning theory and algorithms / Prediction and decision losses

Is Function Similarity Over-Engineered? Building a Benchmark

NeurIPS 2024 · Learning theory and algorithms › Prediction and decision losses · approach: Adaptive testing and experiments · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds a binary function similarity detection benchmark (Assemblage, MOTIF, Common Libraries, Marcelli Dataset-1, BinaryCorp) with leakage-aware splits and proposes REFuSe, a raw-byte MalConv-style CNN trained with triplet loss, which matches or beats GNN, jTrans, and BSim on most datasets.

Problem

Binary function similarity detection (BFSD) benchmarks are small, Linux-centric, and contain data leakage, duplicate functions, unclear labeling, and unrealistic pool-size restrictions, so reported progress does not reflect real-world reverse engineering and malware analysis use.

Approach

Address six dataset construction flaws (train/test leakage, cross-binary duplicates, labeling ambiguity, disabling optimizations, lack of large Windows data, pool-size restrictions) with a new large Windows-centric benchmark, and show that a simple raw-byte CNN baseline (REFuSe) trained with triplet loss is competitive with or superior to feature-engineered models.

Key results

• REFuSe reaches MRR of about 0.61 on Assemblage versus 0.26 for jTrans and 0.26 for the GNN, and is top on four of five datasets (e.g., 0.684 on Common Libraries, 0.676 on Marcelli Dataset-1).
• On BinaryCorp, jTrans (0.693 MRR) outperforms REFuSe (0.468), but jTrans was evaluated with a pool size restriction of 10,000 while REFuSe searched the full 4.79M set.
• Label normalization choice changes REFuSe MRR by 0.63 between the narrowest and loosest labeling schemes.

Measured effects reported in the paper

Surprising findings

• A simple raw-byte CNN outperforms feature-engineered and transformer-based BFSD models on most datasets.
• jTrans performance drops steadily as pool size grows, e.g., MOTIF MRR falls from 0.12 at pool size 100 to 0.06 at 5000.

Limitations

• Labeling criteria are heuristic and based on manual inspection, and the paper calls for future refinement.
• Function similarity is treated as binary same/different rather than a spectrum.
• jTrans fine-tuning used only a 25M-function sample of Assemblage due to compute limits.

Experimental setup

Datasets: Assemblage BinaryCorp Marcelli Dataset-1 MOTIF Common Libraries

Benchmarks: Assemblage MOTIF Common Libraries Marcelli Dataset-1 BinaryCorp

Metrics: mean reciprocal rank recall@k

Baselines: GNN (Li et al. 2019 graph matching network) jTrans BSim Naive Transformer

Builds on: MalConv triplet loss Siamese network Assemblage

Closest papers by meaning

• Assemblage: Automatic Binary Dataset Construction for Machine Learning (NeurIPS 2024 · similarity 0.86)
• Decompile-Bench: Million-Scale Binary-Source Function Pairs for Real-World Binary Decompilation (NeurIPS 2025 · similarity 0.83)
• RustMizan: A Compilable, Contamination-Aware Benchmarking Framework for Rust Vulnerabilities (NeurIPS 2026 · similarity 0.81)
• Source Code Foundation Models are Transferable Binary Analysis Knowledge Bases (NeurIPS 2024 · similarity 0.80)
• Can LLMs Obfuscate Code? A Systematic Analysis of Large Language Models into Assembly Code Obfuscation (AAAI 2025 · similarity 0.80)
• MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents (NeurIPS 2026 · similarity 0.80)