Research map / Large language models / Agents, code and math
HPC-Bench: A Comprehensive Benchmark for High Performance Computing Codes
Bowen Cui, Junyu Yin, Tejas Ramesh, Leo Lim, Oscar Hernandez, Keren Zhou
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces HPC-Bench, a benchmark for evaluating LLMs on high-performance computing code, targeting performance optimization beyond functional correctness.
Problem
Existing code benchmarks evaluate functional correctness and do not capture HPC performance optimization challenges such as low-level code, memory access patterns, and computational kernels.
Approach
A benchmark specifically designed for HPC code performance optimization, rather than only functional correctness of generated code.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| raise required speedup threshold p_thr from 1.0 to 1.5 (ablation) | LLM serial CPU optimization · HPC-Bench EX1 | Fast@3: Claude 0.72 -> 0.17; GPT-5.1 0.69 -> 0.17; Qwen 0.67 -> 0.10 | Hurt |
| raise required speedup threshold p_thr from 1.0 to 1.5 (ablation) | LLM OpenMP parallelization (16 threads) · HPC-Bench EX2 | Fast@3: Qwen 0.27 -> 0.15 | Hurt |
| raise p_thr from 1.0 to 1.5 (ablation) | LLM OpenMP parallelization · HPC-Bench EX2 | Speedup@3: Claude 2.70x -> 2.60x | No clear effect |
| raise p_thr from 1.0 to 1.5 (ablation) | LLM CUDA optimization · HPC-Bench EX3 | Fast@3: 0.45-0.54 -> about 0.09-0.11 | Hurt |
| raise p_thr from 1.0 to 1.5 (ablation) | LLM serial CPU optimization · HPC-Bench EX1 | Speedup@3: Claude 1.46x -> 0.87x | Hurt |
| move from serial CPU to OpenMP to CUDA settings | LLMs · HPC-Bench | fraction of correct trials: 76.9% -> 61.4% -> 38.8% | Hurt |
| increase sampling budget k (ablation) | LLM serial CPU optimization · HPC-Bench EX1 medium | Fast@k | No clear effect |
| increase sampling budget k (ablation) | LLM serial CPU optimization · HPC-Bench EX1 medium | Speedup@k | Improved |
| higher structural complexity stratum | LLM serial CPU optimization · HPC-Bench EX1 | Fast@3 / Speedup@3 | Hurt |
| simple (d1) kernel for OpenMP parallelization | LLM OpenMP parallelization · HPC-Bench EX2 medium 16 threads | fraction reaching 1.05x speedup: 41 of 61 d1 benchmarks fail | Hurt |
| data-parallel, compute-heavy motifs vs irregular motifs | LLM OpenMP parallelization · HPC-Bench EX2 | Speedup@3: NBM 7.17x vs SLA near zero | Mixed |
Experimental setup
Benchmarks: HPC-Bench
Closest papers by meaning
- SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads? (ICML 2026 · similarity 0.88)
- How efficient is LLM-generated code? A rigorous & high-standard benchmark (ICLR 2025 · similarity 0.88)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.87)
- Mercury: A Code Efficiency Benchmark for Code Large Language Models (NeurIPS 2024 · similarity 0.87)
- CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery (ICLR 2025 · similarity 0.87)
- EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
