Research map / Multi-agent systems and games / LLM agents
Outrunning LLM Cutoffs: A Live Kernel Crash Resolution Benchmark for All
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Live-kBench, a self-evolving benchmark that continuously ingests fresh Syzbot Linux kernel crashes, plus kEnv, an agent-agnostic execution environment, and evaluates mini-SWE-agent, SWE-agent, and OpenHands on 534 bugs.
Problem
Static kernel crash-repair benchmarks are vulnerable to data contamination from LLM knowledge cutoffs and do not reflect the evolving Linux kernel, and kernel patch evaluation requires heavy, agent-specific execution setups.
Approach
A live benchmark that weekly scrapes newly fixed Syzbot kernel bugs, filters for reproducibility, and runs agents in a standardized, decoupled kernel execution environment (kEnv over kGymSuite), with time-aware evaluation split at LLM knowledge cutoffs and LLM-judged patch equivalence to developer fixes.
Key results
- Agents achieve up to 25% higher equivalent patch rate on bugs fixed before the LLM knowledge cutoff (Gemini 3 Pro, Pass@1 EPR 15.60% vs 12.97%).
- Mini-SWE-agent with Gemini 3 Pro resolves about 75% of crashes (Mean@3) but only about 15% of patches are equivalent to developer fixes.
- Crash resolution feedback raises mean CRR from 58.11% to 75.03% (29.12% relative) with EPR roughly unchanged.
Surprising findings
- Crash resolution feedback improves crash resolution rate substantially but leaves equivalent patch rate and localization IoU essentially unchanged, suggesting crash suppression rather than genuine repair.
- Perfect file localization (oracle mode) lowers CRR by about 6% while raising EPR by about 12%.
Limitations
- Crash resolution is only a necessary condition for patch validity and equivalence only a sufficient one, so true validity is not guaranteed.
- Main experiments limited to a few closed-source LLMs and three scaffolds due to resource constraints.
- Pre/post-cutoff contamination comparison is confounded by small sample sizes and the single cutoff date used.
Experimental setup
Datasets: Syzbot kernel bugs kBenchSyz Live-kBench-2512
Benchmarks: Live-kBench-2512 kBenchSyz
Metrics: crash resolution rate equivalent patch rate IoU file level IoU function level pass@k mean@k
Baselines: SWE-agent OpenHands CrashFixer kBenchSyz
Builds on: SWE-agent mini-SWE-agent OpenHands kGymSuite kBenchSyz SWE-bench
Closest papers by meaning
- kGym: A Platform and Dataset to Benchmark Large Language Models on Linux Kernel Crash Resolution (NeurIPS 2024 · similarity 0.91)
- MigGPT: Harnessing Large Language Models for Automated Migration of Out-of-Tree Linux Kernel Patches Across Versions (NeurIPS 2025 · similarity 0.86)
- Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agents (ICML 2026 · similarity 0.83)
- DéjàVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving (ICML 2024 · similarity 0.83)
- RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades (NeurIPS 2026 · similarity 0.83)
- ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workload (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.