Research map / Large language models / Agents, code and math
Ekka: Automated Diagnosis of Silent Errors in LLM Inference
Yile Gu, Zhen Zhang, Shaowei Zhu, Xinwei Fu, Jun Wu, Yida Wang, Baris Kasikci
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add mapping validation with error feedback to component mapping (ablation) | Ekka component mapping · 9-bug subset | mapping accuracy: +21.7% | Improved |
| add class-definition retrieval tool to component mapping (ablation) | Ekka component mapping · 9-bug subset | mapping accuracy: +7.6% | Improved |
| add codebase analysis tool and incremental mapping (ablation) | Ekka · 9-bug subset | pass@1/pass@5: 0.67/0.77 -> 0.84/0.88 | Improved |
| add mapping validation/refinement loop (ablation) | Ekka · 9-bug subset | pass@1/pass@5: 0.47/0.66 -> 0.84/0.88 | Improved |
| add code validation and error feedback to activation alignment (ablation) | Ekka activation alignment · 9-bug subset | alignment accuracy: +54.9% | Improved |
| add helper tools and knowledge base to activation alignment (ablation) | Ekka activation alignment · 9-bug subset | alignment accuracy: +26% | Improved |
| add helper tools and knowledge base for alignment (ablation) | Ekka · 9-bug subset, ground-truth mapping | pass@1/pass@5: 0.39/0.55 -> 0.84/0.88 | Improved |
| add alignment validation/refinement loop (ablation) | Ekka · 9-bug subset, ground-truth mapping | pass@1/pass@5: 0.69/0.77 -> 0.84/0.88 | Improved |
| raise error-ratio threshold from 1.5 to 2.0 or 2.5 (ablation) | Ekka error analysis · bug benchmark | pass@1/pass@5: pass@1 0.84 -> 0.88 at 2.5; pass@5 0.88 unchanged | No clear effect |
| use Ekka differential debugging instead of general coding agents | Ekka vs Mini-SWE-Agent and OpenCode · silent error benchmark, pass@1 | diagnosis accuracy improvement: +28% over Mini-SWE-Agent, +34% over OpenCode | Improved |
Closest papers by meaning
- The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility (NeurIPS 2026 · similarity 0.85)
- RvLLM: LLM Runtime Verification with Domain Knowledge (NeurIPS 2025 · similarity 0.85)
- Precise Debugging Benchmark: Is Your Model Debugging or Regenerating? (NeurIPS 2026 · similarity 0.85)
- CoRe: Benchmarking LLMs’ Code Reasoning Capabilities through Static Analysis Tasks (NeurIPS 2025 · similarity 0.84)
- Teaching Large Language Models to Self-Debug (ICLR 2024 · similarity 0.84)
- kGym: A Platform and Dataset to Benchmark Large Language Models on Linux Kernel Crash Resolution (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
