Research map / Large language models / Language, safety and interpretability

Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates

ICLR 2025 · Large language models › Language, safety and interpretability · approach: Safety alignment of LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that a "null model" returning a constant, input-irrelevant response, crafted with a structured cheating template plus a transferable random-search prefix, can reach top-ranked win rates on AlpacaEval 2.0, Arena-Hard-Auto and MT-Bench judged by GPT-4 and Llama-3 auto-annotators.

Problem

Automatic LLM benchmarks that use LLM auto-annotators to compare model outputs can be gamed; the paper asks whether a model that ignores the instructions and outputs constant text can achieve top win rates when the benchmark instructions are private.

Approach

Replace the model response with a structured counterfeit evaluation-template fragment that is resilient to position swaps, prepend a transferable adversarial prefix optimized by universal random search on public UltraFeedback instructions and GPT-4 judge losses, and wrap it as a constant-output null model.

Key results

• Structured+RS null model achieves 86.5% LC win rate and 76.9% raw win rate on AlpacaEval 2.0 with GPT-4-1106-Preview judge, above the verified SOTA at 57.5% LC.
• On Arena-Hard-Auto the Structured+RS cheat scores 83.0 win rate versus 82.6 for the verified SOTA; on MT-Bench it scores 9.55 versus 8.96.
• Against Llama-3-8B-Instruct and Llama-3-70B-Instruct judges, random search lifts LC win rate from under 3% to 95.4% and from 0.4% to 95.1% respectively.

Measured effects reported in the paper

Surprising findings

• A structured response with no informative content reaches 76.8% LC win rate on AlpacaEval 2.0 with GPT-4 judge, while plain persuasive null responses get under 1%.
• The cheats transfer poorly to GPT-3.5-Turbo-1106 as judge (0.4% LC win rate for Structured+RS), and a windowed PPL filter fails to flag the structured responses.

Limitations

• Experiments are proof-of-concept and rely on manually crafted structured responses.
• Cheats transfer poorly across judge models such as GPT-3.5-Turbo.
• Evaluated only on specific benchmarks; generality to other less-studied benchmarks is uncertain.

Experimental setup

Datasets: AlpacaEval 2.0 test set (805 instructions) Arena-Hard-Auto MT-Bench UltraFeedback AlpacaEval human annotations (2.5K)

Benchmarks: AlpacaEval 2.0 Arena-Hard-Auto MT-Bench

Metrics: length-controlled win rate raw win rate discrete win rate 95% CI win rate MT-Bench score human agreement Spearman correlation Pearson correlation

Baselines: Verified SOTA Community SOTA GPT-3.5 Turbo GCG SmoothLLM Self-Reminder PPL filter

Builds on: AlpacaEval 2.0 Arena-Hard-Auto MT-Bench universal random search SmoothLLM

Closest papers by meaning

• LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming? (NeurIPS 2025 · similarity 0.87)
• LiveBench: A Challenging, Contamination-Limited LLM Benchmark (ICLR 2025 · similarity 0.87)
• Rethinking Verification for LLM Code Generation: From Generation to Testing (NeurIPS 2025 · similarity 0.86)
• OR-Bench: An Over-Refusal Benchmark for Large Language Models (ICML 2025 · similarity 0.86)
• Training on the Benchmark Is Not All You Need (AAAI 2025 · similarity 0.86)
• Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference (NeurIPS 2025 · similarity 0.86)