Research map / Large language models / Agents, code and math
Nova: Generative Language Models for Assembly Code with Hierarchical Attention and Contrastive Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Nova, a generative LLM for x86-64 assembly that adds hierarchical self-attention and functionality/optimization contrastive objectives on top of DeepSeek-Coder, improving binary decompilation and binary code similarity detection.
Problem
Generative LLMs pre-trained on source code do not transfer well to assembly code, which has low information density (long token sequences per semantic unit) and high syntactic diversity across compiler optimization levels.
Approach
Hierarchical self-attention with per-instruction [INST] summary tokens (intra-instruction, preceding-instruction, and inter-instruction attention applied to half the heads), combined with functionality and optimization contrastive losses that align same-function embeddings across forms and order embeddings by optimization level.
Key results
- On HumanEval-Decompile, Nova-1.3B reaches 25.17 average Pass@1 and 38.30 Pass@10, versus 10.33 and 16.72 for LLM4Decompile-1.3B (+14.84 and +21.58 points).
- Nova-6.7B reaches 34.36 average Pass@1 and 46.91 Pass@10, outperforming LLM4Decompile-6.7B by 12.41 and 15.53 points.
- On BCSD with K=50, Nova-6.7B averages 0.86 Recall@1 versus 0.81 for CodeArt and 0.67 for jTrans; Nova-1.3B ties CodeArt at K=500 (0.64).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add contrastive learning objectives (ablation) | DeepSeek-Coder-1.3B assembly model without hierarchical attention · HumanEval-Decompile | Pass@1: 15.78 -> 21.86 | Helped |
| Add hierarchical self-attention (ablation) | Nova-1.3B · HumanEval-Decompile | Pass@1: 21.86 -> 25.17 | Helped |
| Add hierarchical self-attention (ablation) | Nova-1.3B · HumanEval-Decompile | Pass@10: 35.25 -> 38.30 | Helped |
| Add functionality contrastive learning (ablation) | DeepSeek-Coder-1.3B assembly model without hierarchical attention or optimization CL · HumanEval-Decompile | Pass@1: 15.78 -> 17.05 | Helped |
| Add optimization contrastive learning (ablation) | DeepSeek-Coder-1.3B assembly model without hierarchical attention or functionality CL · HumanEval-Decompile | Pass@1: 15.78 -> 20.40 | Helped |
| Apply hierarchical attention to all attention heads instead of half (ablation) | Nova-1.3B · HumanEval-Decompile | Pass@1: 25.17 -> 22.04 | Mixed |
| Replace hierarchical attention with LongCoder attention (ablation) | Nova-1.3B · HumanEval-Decompile | Pass@1: 25.17 -> 22.68 | Hurt |
| Apply Nova's approaches (hierarchical attention and contrastive learning) instead of standard fine-tuning | Granite-3B-Code-128K · HumanEval-Decompile | Pass@1: 13.97 -> 18.49 | Helped |
| Apply Nova's approaches (hierarchical attention and contrastive learning) instead of standard fine-tuning | Granite-3B-Code-128K · HumanEval-Decompile | Pass@10: 20.98 -> 25.60 | Helped |
| Use Nova-1.3B instead of LLM4Decompile-1.3B | DeepSeek-Coder-1.3B · HumanEval-Decompile | Pass@1: 10.33 -> 25.17 | Helped |
| Use Nova-1.3B instead of LLM4Decompile-1.3B | DeepSeek-Coder-1.3B · HumanEval-Decompile | Pass@10: 16.72 -> 38.30 | Helped |
| Increase model size from Nova-1.3B to Nova-6.7B | Nova · BCSD average (K=50, 100, 200, 500) | Recall@1 | Helped |
Surprising findings
- Applying hierarchical attention to all attention heads did not improve over half-head application and sometimes dropped performance, suggesting full self-attention retains knowledge hierarchical attention misses.
- Nova-1.3B outperforms GPT-4o in average Pass@1 and Pass@10 on HumanEval-Decompile despite being far smaller.
Limitations
- Only X86-64 assembly for C programs is collected and modeled, so results are not shown for other architectures such as ARM.
- Only 1.3B and 6.7B model scales are developed due to compute limits; larger scales are untested.
- Pre-training data was subsampled (e.g., 2.16M BCD pairs, 138.8K of 4M compiled The-Stack programs) due to computation constraints.
Experimental setup
Datasets: AnghaBench The-Stack BinaryCorp-3M HumanEval-Decompile
Benchmarks: HumanEval-Decompile Binutils Curl ImageMagick SQLite OpenSSL Putty
Metrics: Pass@1 Pass@10 Recall@1
Baselines: LLM4Decompile-1.3B LLM4Decompile-6.7B CodeLlama-7B StarCoder2-7B DeepSeekCoder-7B Qwen-2.5-Coder-7B LLMCompiler-7B GPT-3.5-Turbo GPT-4o jTrans DiEmph CodeArt
Builds on: DeepSeek-Coder LongCoder LLM4Decompile CodeArt jTrans
Closest papers by meaning
- ASAP: Assembly-Source Aligned Pseudocode Refinement For Binary Decompilation (NeurIPS 2026 · similarity 0.86)
- Uncovering LLM-Generated Code: A Zero-Shot Synthetic Code Detector via Code Rewriting (AAAI 2025 · similarity 0.85)
- Guess & Sketch: Language Model Guided Transpilation (ICLR 2024 · similarity 0.85)
- EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning (ICML 2025 · similarity 0.85)
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination (ICML 2025 · similarity 0.85)
- SelfCodeAlign: Self-Alignment for Code Generation (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.