Research map / Large language models / Agents, code and math
ProbeLLM: Automating Principled Diagnosis of LLM Failures
Yue Huang, Zhengzhe Jiang, Yuchen Ma, Yu Jiang, Xiangqi Wang, Yujun Zhou, Yuexing Hao, Kehan Guo, Pin-Yu Chen, Marzyeh Ghassemi, Stefan Feuerriegel, Xiangliang Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ProbeLLM hierarchical MCTS probing instead of AutoDetect | Llama-3.1-8b-ins. target · MMLU | error rate (%): 67% -> 86% | Improved |
| use ProbeLLM instead of PAIR | Llama-3.1-8b-ins. target · MMLU | error rate (%): 46% -> 86% | Improved |
| use ProbeLLM instead of AutoDetect on SuperGLUE seeds | Llama-3.1-8b-ins. target · SuperGLUE | error rate (%): 38% -> 71% | Improved |
| use ProbeLLM instead of PAIR or AutoDetect for clustering failures | Llama-3.1-8b-ins. target · failure clusters | noise rate (%): 60.9 -> 37.0 (AutoDetect), 77.2 -> 37.0 (PAIR) | Improved |
| probe with ProbeLLM instead of static benchmarks | Llama-3.1-8b-ins. target · failure mode discovery | number of clusters: 4 -> 19 | Improved |
| increase MCTS search depth (ablation) | ProbeLLM, 12 target models · MMLU, MBPP, HellaSwag, TruthfulQA | error rate (%) | Improved |
| increase MCTS search depth to 4 on SuperGLUE (ablation) | ProbeLLM · SuperGLUE | error rate (%) | Hurt |
| use tool-augmented generation (Python exec, web search) instead of perturbation only (ablation) | ProbeLLM · tool usage analysis | error rate (%): 45.0% -> 60.7% | Improved |
| vary MACRO/MICRO simulation ratio (beta) (ablation) | ProbeLLM · five seed benchmarks | error rate (%) | No clear effect |
| fine-tune on ProbeLLM-discovered failure data (100 to 250 samples) | Llama-3.1-8b-ins. and GPT-oss-20b · error-prone evaluation set | performance (%) | Mixed |
| fine-tune on ProbeLLM-discovered failure data | Llama-3.1-8b-ins. and GPT-oss-20b · held-out existing benchmark subsets | performance (%) | No clear effect |
| vary HDBSCAN clustering hyperparameters (ablation) | ProbeLLM failure modes · clustering stability | cluster stability | No clear effect |
Closest papers by meaning
- Language Models Can Predict Their Own Behavior (NeurIPS 2025 · similarity 0.88)
- Fast and Accurate Probing of In-Training LLMs' Downstream Performances (NeurIPS 2026 · similarity 0.87)
- Context-Aware Testing: A New Paradigm for Model Testing with Large Language Models (NeurIPS 2024 · similarity 0.87)
- Detecting High-Stakes Interactions with Activation Probes (NeurIPS 2025 · similarity 0.87)
- RvLLM: LLM Runtime Verification with Domain Knowledge (NeurIPS 2025 · similarity 0.87)
- Less Diverse, Less Safe: The Indirect But Pervasive Risk of Test-Time Scaling in Large Language Models (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
