AI Paper Atlas

Research map / Large language models / Agents, code and math

HPC-Bench: A Comprehensive Benchmark for High Performance Computing Codes

Bowen Cui, Junyu Yin, Tejas Ramesh, Leo Lim, Oscar Hernandez, Keren Zhou

NeurIPS 2026 · Large language models › Agents, code and math · approach: Execution-based code verification · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces HPC-Bench, a benchmark for evaluating LLMs on high-performance computing code, targeting performance optimization beyond functional correctness.

Problem

Existing code benchmarks evaluate functional correctness and do not capture HPC performance optimization challenges such as low-level code, memory access patterns, and computational kernels.

Approach

A benchmark specifically designed for HPC code performance optimization, rather than only functional correctness of generated code.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
raise required speedup threshold p_thr from 1.0 to 1.5 (ablation)LLM serial CPU optimization · HPC-Bench EX1Fast@3: Claude 0.72 -> 0.17; GPT-5.1 0.69 -> 0.17; Qwen 0.67 -> 0.10Hurt
raise required speedup threshold p_thr from 1.0 to 1.5 (ablation)LLM OpenMP parallelization (16 threads) · HPC-Bench EX2Fast@3: Qwen 0.27 -> 0.15Hurt
raise p_thr from 1.0 to 1.5 (ablation)LLM OpenMP parallelization · HPC-Bench EX2Speedup@3: Claude 2.70x -> 2.60xNo clear effect
raise p_thr from 1.0 to 1.5 (ablation)LLM CUDA optimization · HPC-Bench EX3Fast@3: 0.45-0.54 -> about 0.09-0.11Hurt
raise p_thr from 1.0 to 1.5 (ablation)LLM serial CPU optimization · HPC-Bench EX1Speedup@3: Claude 1.46x -> 0.87xHurt
move from serial CPU to OpenMP to CUDA settingsLLMs · HPC-Benchfraction of correct trials: 76.9% -> 61.4% -> 38.8%Hurt
increase sampling budget k (ablation)LLM serial CPU optimization · HPC-Bench EX1 mediumFast@kNo clear effect
increase sampling budget k (ablation)LLM serial CPU optimization · HPC-Bench EX1 mediumSpeedup@kImproved
higher structural complexity stratumLLM serial CPU optimization · HPC-Bench EX1Fast@3 / Speedup@3Hurt
simple (d1) kernel for OpenMP parallelizationLLM OpenMP parallelization · HPC-Bench EX2 medium 16 threadsfraction reaching 1.05x speedup: 41 of 61 d1 benchmarks failHurt
data-parallel, compute-heavy motifs vs irregular motifsLLM OpenMP parallelization · HPC-Bench EX2Speedup@3: NBM 7.17x vs SLA near zeroMixed

Experimental setup

Benchmarks: HPC-Bench

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy