Research map / Large language models / Language, safety and interpretability
When LLMs get significantly worse: A statistical approach to detect model degradations
Jonas M. Kübler, Kailash Budhathoki, Matthäus Kleindessner, Xiong Zhou, Junming Yin, Ashish Khetan, George Karypis
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Compare per-sample scores with exact one-sided McNemar test instead of treating accuracies as independent | degradation detection · Llama-3.1 8B Instruct KV-FP8 | detection of 0.79% drop: p = 1.69e-05 detects drop | Improved |
| Use the McNemar-based tests on KV-FP8 for the 70B model | exact McNemar tests · Llama-3.3 70B Instruct KV-FP8 | detection: 0.29% drop, p_pool 2.51e-02 | Improved |
| Apply tests to theoretically lossless variants (rerun, transformers, TP1, A100) | exact McNemar tests · Llama-3.1 8B Instruct | false positives: p_pool 5.64e-01 to 9.21e-01 | Improved |
| Use hard accuracy-difference threshold (>2%) as degradation rule | threshold baseline · Llama-3.1 8B Instruct KV-FP8 | detection: 0.79 vs 2% threshold | Hurt |
| Use hard flip-probability threshold (>=5%) as degradation rule | flip-rate baseline · Llama-3.1 8B Instruct FP8 | false positives: 8.71% flips vs 9.03% for KV-FP8 | Hurt |
| Remove examples that never flip from MMLU-Pro (ablation) | degradation testing · Llama-3.1 8B Instruct MMLU-Pro | flip rate KV-FP8: 15.49% -> 24.42% | Improved |
| Use w4a16 INT4 quantization | Llama-3.1 8B Instruct · Open LLM Leaderboard v2 | pooled accuracy: 42.43 -> 40.70 | Hurt |
| Use FP8 weights/activations | Llama-3.1 8B Instruct · Open LLM Leaderboard v2 | pooled accuracy: 42.43 -> 42.48 | No clear effect |
| Use 2:4 sparsity (pruned and distilled) | Llama-3.1 8B base · Open LLM Leaderboard v2 | pooled accuracy: 32.72 -> 30.13 | Hurt |
Closest papers by meaning
- Rethinking LLM Evaluation: Can We Evaluate LLMs with 200× Less Data? (ICLR 2026 · similarity 0.88)
- The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility (NeurIPS 2026 · similarity 0.87)
- When Models Don’t Collapse: On the Consistency of Iterative MLE (NeurIPS 2025 · similarity 0.86)
- Model Collapse Demystified: The Case of Regression (NeurIPS 2024 · similarity 0.86)
- Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference (NeurIPS 2025 · similarity 0.86)
- Statistical Hypothesis Testing for Auditing Robustness in Language Models (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
