Research map / Large language models / Agents, code and math
kGym: A Platform and Dataset to Benchmark Large Language Models on Linux Kernel Crash Resolution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces kGym, a parallel VM-based platform for building and running Linux kernels, and kBenchSyz, a 279-bug Linux kernel crash-resolution benchmark, on which LLMs achieve very low fix rates.
Problem
LLMs are evaluated mostly on isolated coding puzzles or userspace repository repair, not on large, concurrent, low-level systems code such as the Linux kernel, where crash resolution requires building and executing kernels to validate patches.
Approach
An execution platform (kGym) that compiles kernel commits and replays Syzkaller reproducers across parallel VMs to automatically verify LLM-generated patches, plus a benchmark (kBenchSyz) of real syzbot kernel bugs with developer fixes.
Key results
- Best unassisted (BM25) solve rate is 0.72% and best oracle-assisted solve rate is 5.38% (GPT-4 Turbo) on 279 bugs.
- Across all models, union of correct patches solves 29 unique bugs (10.39% combined solve rate).
- Over 17,000 kernel jobs were run on kGym; patch apply rates for GPT-4 Turbo reach about 55-57%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use oracle retrieval instead of BM25 | GPT-4 Turbo, Top-10 · kBenchSyz | solve rate (%): 0.72 -> 5.38 | Helped |
| Use oracle retrieval instead of BM25 | GPT-3.5 Turbo, Top-10 · kBenchSyz | solve rate (%): 0.36 -> 1.08 | Helped |
| Use oracle retrieval instead of BM25 | Gemini-1.5 Pro, Top-10 · kBenchSyz | solve rate (%): 0 -> 3.58 | Helped |
| Use oracle retrieval instead of BM25 | Claude-3 Sonnet, Top-1 · kBenchSyz | solve rate (%): 0 -> 1.79 | Helped |
| Increase the number of generated patches from Top-1 to Top-10 | GPT-4 Turbo with Oracle retrieval · kBenchSyz | solve rate (%): 1.08 -> 5.38 | Helped |
| Increase the number of generated patches from Top-1 to Top-10 | GPT-3.5 Turbo with Oracle retrieval · kBenchSyz | solve rate (%): 0 -> 1.08 | Helped |
| Increase the number of generated patches from Top-1 to Top-10 | Gemini-1.5 Pro with Oracle retrieval · kBenchSyz | solve rate (%): 0.72 -> 3.58 | Helped |
| Increase the number of generated patches from Top-1 to Top-10 | GPT-3.5 Turbo with BM25 retrieval · kBenchSyz | solve rate (%): 0.36 -> 0.36 | No clear effect |
Surprising findings
- Open-source Code Llama and Llama-3 models solve 0% of bugs in all settings.
- BM25 retrieval recall of oracle files is very low (under 10% even at top-20), limiting unassisted performance.
Limitations
- Kernel compilation and reproducer runs are slow, giving 17-30 minute feedback per patch.
- A single reproducer test may pass while the patch still alters functionality, so extra testing is needed.
- Context limits (16K/50K) force skipping many bugs whose oracle files do not fit in the prompt.
Experimental setup
Datasets: Syzbot kernel bug reports Syzkaller reproducers
Benchmarks: kBenchSyz
Metrics: patch apply rate bug solve rate bug localization overlap
Baselines: GPT-3.5 Turbo GPT-4 Turbo Claude-3 Sonnet Gemini-1.5 Pro Code Llama 7B Code Llama 13B Code Llama 34B Llama-3-8B
Builds on: Syzkaller SWE-bench
Closest papers by meaning
- Outrunning LLM Cutoffs: A Live Kernel Crash Resolution Benchmark for All (ICML 2026 · similarity 0.91)
- MigGPT: Harnessing Large Language Models for Automated Migration of Out-of-Tree Linux Kernel Patches Across Versions (NeurIPS 2025 · similarity 0.88)
- CodeChemist: Test-Time Scaling for Low-Resource Code Generation via Functional Knowledge Transfer (ICML 2026 · similarity 0.86)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.86)
- Teaching Large Language Models to Self-Debug (ICLR 2024 · similarity 0.86)
- SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads? (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.