Research map / Large language models / Agents, code and math

CUDABench: Benchmarking LLMs for Text-to-CUDA Generation

NeurIPS 2026 · Large language models › Agents, code and math · approach: Code generation benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces CUDABench, a text-to-CUDA benchmark of 500 tasks (1,500 prompts across three difficulty levels) with a roofline-based Performance-Score, and evaluates seven frontier LLMs on it.

Problem

Existing GPU kernel benchmarks mostly translate PyTorch or other high-level code into CUDA and rely on execution time, so they do not measure text-to-CUDA generation or hardware-independent performance.

Approach

Combines a Breadth-Depth-Difficulty task space with execution-based verification and a roofline-based Performance-Score (achieved GFLOPs/s divided by attainable GFLOPs/s), aggregated into CUDABench-Score over compile- and functionally-correct kernels.

Key results

• Claude 4.5 Sonnet reaches 99.8% compilation and 85.8% functional correctness at Level 1 (Pass@1), while GPT-5.2 (High) has the top Level 1 CUDABench-Score of 40.9%.
• Functional correctness falls from 79.8% (GPT-5.2, Level 1) to 60.8% (Level 3) at Pass@1.
• On the Scientific Simulation & Finance domain at Level 3, Claude 4.5 Sonnet has an 85.0% failure rate.

Surprising findings

• High compilation rates (often above 90%) coexist with much lower functional correctness, so most failures are functional rather than syntactic.
• Top models score nearly identically on CUDABench-Score (about 40%), suggesting the performance bottleneck is systemic rather than model-specific.

Limitations

• Main experiments run on a single NVIDIA A40 GPU; the RTX 4090 check covers only four models.
• Performance-Score is limited to kernels that are both compiled and functionally correct, and the paper's roofline-based metric is evaluated mainly on FP32 kernels.

Experimental setup

Datasets: CUDABench-Set

Benchmarks: CUDABench-Set

Metrics: compilation success functional consistency Performance-Score CUDABench-Score pass@1 pass@3

Baselines: KernelBench TritonBench TritonGym BabelTower ComputeEval

Builds on: KernelBench roofline model pass@k

Closest papers by meaning

• Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs (ICML 2025 · similarity 0.88)
• KernelBench: Can LLMs Write Efficient GPU Kernels? (ICML 2025 · similarity 0.87)
• Practical Offloading for Fine-Tuning LLM on Commodity GPU via Learned Sparse Projectors (AAAI 2025 · similarity 0.87)
• A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints (ICML 2026 · similarity 0.87)
• BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Models (NeurIPS 2024 · similarity 0.86)
• EasySpec: Layer-Parallel Speculative Decoding for Efficient Multi-GPU Utilization (NeurIPS 2025 · similarity 0.86)