Research map / Large language models / Instruction and fine-tuning
ComBack: A Versatile Dataset for Enhancing Compiler Backend Development Efficiency
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ComBack, a dataset of 178 GCC/LLVM compiler backends with three tasks (statement completion, next-statement suggestion, code generation), and shows fine-tuned small code models, especially CodeT5+ (220M), beat Fork-Flow, ChatGPT-3.5-Turbo and Code-LLaMA-34B-Instruct on new targets.
Problem
Compiler backend development for new processor targets is laborious and manual, and language models lack public training data for this domain.
Approach
A public, multi-target dataset of compiler backend functions from GCC and LLVM, with target-specific values abstracted into placeholder tokens and three derived tasks, used to fine-tune code language models for backend code completion and generation.
Key results
- Fine-tuning improves six models: CodeT5+ reaches 66.93% EM-style accuracy band on statement completion ED 84.45 and BLEU-4 75.29 on code generation after fine-tuning.
- On RISC-V, ARC and NVPTX, fine-tuned CodeT5+ beats ChatGPT-3.5-Turbo by 37.10%-40.82% EM on statement completion and Code-LLaMA-34B-Instruct by 49.72%-54.57%.
- Fine-tuned CodeT5+ exceeds both average and maximum Fork-Flow edit distance and BLEU-4 for code generation on new targets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune CodeT5+ on ComBack | CodeT5+ · Statement-Level Completion | EM (%): 0.02 -> 66.93 | Helped |
| Fine-tune CodeT5+ on ComBack | CodeT5+ · Next-Statement Suggestion | EM (%): 0.12 -> 59.57 | Helped |
| Fine-tune CodeT5+ on ComBack | CodeT5+ · Code Generation | BLEU-4: 0.00 -> 75.29 | Helped |
| Fine-tune CodeT5+ on ComBack | CodeT5+ · GCC RISC-V, ARC, and NVPTX Code Generation | BLEU-4: CodeT5+: 32.56, 19.94, 25.47; ChatGPT: 1.37, 1.67, 1.57 | Helped |
| Train with MPU and GPU data rather than CPU-only data (ablation) | CodeT5+ · GCC ARC (MPU) Code Generation | BLEU-4: 19.29 -> 19.94 | Helped |
| Train with MPU and GPU data rather than CPU-only data (ablation) | CodeT5+ · GCC NVPTX (GPU) Code Generation | BLEU-4: 22.46 -> 25.47 | Helped |
| Train with MPU and GPU data rather than CPU-only data (ablation) | CodeT5+ · LLVM ARC (MPU) Code Generation | BLEU-4: 49.62 -> 55.38 | Helped |
| Train with MPU and GPU data rather than CPU-only data (ablation) | CodeT5+ · LLVM NVPTX (GPU) Code Generation | BLEU-4: 42.94 -> 44.33 | Helped |
| Add RISC-V training data (ablation) | CodeT5+ · RI5CY Statement-Level Completion | EM (%): +7.90 | Helped |
| Add RISC-V training data (ablation) | CodeT5+ · RI5CY Next-Statement Suggestion | EM (%): +9.96 | Helped |
| Add RISC-V training data (ablation) | CodeT5+ · RI5CY Code Generation | BLEU-4: +25.05 | Helped |
Surprising findings
- Fine-tuned CodeT5+ with 220M parameters outperforms ChatGPT-3.5-Turbo and Code-LLaMA-34B-Instruct on the backend tasks.
- Training only on CPU targets gives reduced accuracy on most MPU and GPU targets, showing limited transfer to new target types.
Limitations
- Highly customized backend functions lack natural language descriptions, so they are excluded from code generation.
- Models trained on CPU-only data transfer poorly to GPU and MPU targets.
Experimental setup
Datasets: ComBack (GCC and LLVM backends from GitHub and official releases)
Benchmarks: ComBack Statement-Level Completion ComBack Next-Statement Suggestion ComBack Code Generation
Metrics: exact match Levenshtein edit distance similarity BLEU-4
Baselines: Fork-Flow ChatGPT-3.5-Turbo Code-LLaMA-34B-Instruct
Builds on: CodeXGlue Fork-Flow
Closest papers by meaning
- Teaching Large Language Models to Self-Debug (ICLR 2024 · similarity 0.85)
- Can Large Language Models Understand Intermediate Representations in Compilers? (ICML 2025 · similarity 0.85)
- CodeChemist: Test-Time Scaling for Low-Resource Code Generation via Functional Knowledge Transfer (ICML 2026 · similarity 0.85)
- Chain of Execution Supervision Promotes General Reasoning in Large Language Models (NeurIPS 2025 · similarity 0.84)
- RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems (ICLR 2024 · similarity 0.84)
- SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads? (ICML 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.