Research map / Large language models / Agents, code and math

KernelBench: Can LLMs Write Efficient GPU Kernels?

ICML 2025 · Large language models › Agents, code and math · approach: LLMs for optimization and heuristics · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces KernelBench, a 250-task benchmark and framework for evaluating whether LLMs can write correct and fast GPU (CUDA) kernels for PyTorch workloads, with a new fast_p metric and analysis of one-shot, repeated sampling, and iterative refinement.

Problem

Efficient GPU kernels are needed for performant ML, but writing them requires scarce expertise. The paper asks whether language models can automatically generate correct and speedup-producing kernels for diverse PyTorch workloads.

Approach

A verifiable benchmark of 250 PyTorch tasks across three levels (single ops, operator sequences, full architectures) where models output a ModelNew class with inline CUDA kernels, scored by a new fast_p metric that counts tasks that are both correct and faster than a threshold p over PyTorch Eager.

Key results

• One-shot OpenAI o1 and DeepSeek-R1 match or beat PyTorch Eager on fewer than 20% of tasks; at fast_1, DeepSeek-R1 reaches 36% on Level 2 with one-shot generation on L40S.
• Iterative refinement with execution and profiler feedback (G+E+P) raises fast_1 from 12%, 36%, 12% to 43%, 72%, 18% across Levels 1-3 for DeepSeek-R1.
• Few-shot in-context examples degrade overall fast_1 for o1 because models attempt more aggressive optimizations that fail to compile or are incorrect.

Measured effects reported in the paper

Surprising findings

• Adding in-context kernel optimization examples lowered overall fast_1 even though correct solutions became more optimized, because of more execution failures.
• Providing hardware specifications in context did not significantly change outputs of Llama 3.1 70B or DeepSeek-V3.

Limitations

• Correctness checked with only 5 random inputs, so equivalence is heuristic rather than formal.
• Evaluation limited to NVIDIA GPUs and CUDA C via inline extensions; most results are on a single L40S GPU.
• Iterative refinement experiments use temperature 0, and DeepSeek-R1 could not be repeatedly sampled because its API lacks a temperature parameter.

Experimental setup

Datasets: KernelBench 250 tasks drawn from PyTorch, huggingface/transformers, and pytorch-image-models repositories

Benchmarks: KernelBench

Metrics: fast_p correctness rate speedup distribution

Baselines: PyTorch Eager torch.compile Llama 3.1-70B Instruct Llama 3.1-405B Instruct GPT-4o Claude 3.5 Sonnet DeepSeek V3

Builds on: torch.compile Triton ThunderKittens

Closest papers by meaning

• CUDABench: Benchmarking LLMs for Text-to-CUDA Generation (NeurIPS 2026 · similarity 0.87)
• EGG: An Expert-Guided Agent Framework for Kernel Generation (ICML 2026 · similarity 0.87)
• Pfeife: Automatic Pipeline Parallelism for PyTorch (ICML 2025 · similarity 0.86)
• kGym: A Platform and Dataset to Benchmark Large Language Models on Linux Kernel Crash Resolution (NeurIPS 2024 · similarity 0.85)
• Cost-Efficient LLM Training with Lifetime-Aware Tensor Offloading via GPUDirect Storage (NeurIPS 2025 · similarity 0.84)
• Practical Offloading for Fine-Tuning LLM on Commodity GPU via Learned Sparse Projectors (AAAI 2025 · similarity 0.84)