Research map / Large language models / Language, safety and interpretability
Beyond Benchmarks: Toward Causally Faithful Evaluation of Large Language Models
Zhengshuyuan Tian, Wanling Gao, Chuanxin Lan, Chenxi Wang, Lei Wang, Guoxin Kang, Zhengxin Yang, Yunyou Huang, Xuehai Hong, Jianfeng Zhan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| vary uncontrolled evaluation components (CoT, temperature, top p, presence penalty, max tokens) over 162 conditions (ablation) | LLM evaluation · AIME | accuracy: up to 70% variation | Mixed |
| evaluate under sampled SCS configurations instead of the default original setting | DeepSeek · AIME'24 | accuracy: 0.367 -> 0.211 | Hurt |
| evaluate under sampled SCS configurations instead of the default original setting | Doubao · AIME'24 | accuracy: 0.030 -> 0.254 | Improved |
| add augmented questions (A-SCS) to the SCS evaluation | Kimi · AIME'24 | accuracy: 0.082 -> 0.235 | Improved |
| add interaction of Shot and Multi Turn (ablation) | GPT-3.5 · evaluation accuracy | significance: p = 0.58, p = 0.87 alone -> p = 0.01 interaction | Improved |
| vary Question Format (ablation) | DeepSeek · AIME'24 ANOVA | variance explained eta^2: 0.399643 | Improved |
| add chain-of-thought prompting (ablation) | DeepSeek · AIME'24 ANOVA | variance explained eta^2: 0.080156 | Improved |
| vary max tokens (ablation) | DeepSeek · AIME'24 ANOVA | variance explained eta^2: 0.028099 | Improved |
| vary temperature and top p jointly (ablation) | DeepSeek · AIME'24 ANOVA | variance explained eta^2: 0.002082, p = 0.178 | No clear effect |
| vary Question Paraphrase and presence penalty (ablation) | GPT-4.1 · AIME'24 | significance: p = 0.19 and 0.16 | No clear effect |
| compare models by interval estimates instead of default configuration | pairwise model comparisons · AIME'24 | ranking agreement: 9 sign reversals, 7 missed differences, 3 illusory gaps (18 cases) | Hurt |
| insert distractor sentences into a question (analogical transformation) | nine LLMs · single seen question | correct solutions: 5 solved -> 0 solved | Hurt |
Closest papers by meaning
- An Empirical Analysis of Uncertainty in Large Language Model Evaluations (ICLR 2025 · similarity 0.91)
- Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.91)
- EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solving (NeurIPS 2025 · similarity 0.91)
- LLMEval: A Preliminary Study on How to Evaluate Large Language Models (AAAI 2024 · similarity 0.90)
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.90)
- tinyBenchmarks: evaluating LLMs with fewer examples (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
