Research map / Reinforcement learning / RL from feedback
On The Fragility of Benchmark Contamination Detection in Reasoning Models
Han Wang, Haoyu Li, Brian Ko, Huan Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| run GRPO with clean data after SFT contamination (Min-K%) (ablation) | SFT-contaminated Qwen2.5-7B · 6 reasoning benchmarks | AUROC (%): 74.96 -> 61.27 | Hurt |
| run GRPO with clean data (LiRA) (ablation) | SFT-contaminated Qwen2.5-7B · 6 reasoning benchmarks | AUROC (%): 89.13 -> 80.14 | Hurt |
| run GRPO with clean data (Max-K%) (ablation) | SFT-contaminated Qwen2.5-7B · 6 reasoning benchmarks | AUROC (%): 69.83 -> 52.35 | Hurt |
| GRPO on clean & member data instead of clean only (ablation) | SFT-contaminated Qwen2.5-7B · 6 reasoning benchmarks | AUROC (%): 61.27 -> 58.54 | Hurt |
| add clipping to RAFT++ objective (ablation) | RAFT++ · 6 reasoning benchmarks | Loss AUROC (%): 74.39 -> 57.58 | Hurt |
| add clipping to GRPO objective (ablation) | GRPO · 6 reasoning benchmarks | Loss AUROC (%): 73.28 -> 61.26 | Hurt |
| plain rejection sampling (RAFT) after SFT contamination (ablation) | SFT-contaminated Qwen2.5-7B · 6 reasoning benchmarks | Loss AUROC (%): 75.48 -> 77.51 | No clear effect |
| SFT contamination on Qwen2.5-7B-Instruct before RL | clean SFT · 6 reasoning benchmarks | pass@1 avg (%): 38.41 -> 47.23 | Improved |
| RL contamination (benchmark questions with rewards) vs clean RL | Qwen2.5-7B SFT · 6 reasoning benchmarks | pass@1 avg (%): 45.55 -> 44.96 | No clear effect |
| extensive SFT contamination with CoT on DS Llama-8B | DeepSeek-R1-Distill-Llama-8B · 6 reasoning benchmarks | pass@1 avg (%): 48.38 -> 60.14 | Improved |
| extensive SFT contamination with CoT on DS Qwen-7B | DeepSeek-R1-Distill-Qwen-7B · 6 reasoning benchmarks | pass@1 avg (%): 54.42 -> 57.10 | Improved |
| detect final-stage CoT contamination (Max-K%) | contaminated DS Qwen-14B · 6 reasoning benchmarks | AUROC (%): 51.32 | No clear effect |
Closest papers by meaning
- ConStat: Performance-Based Contamination Detection in Large Language Models (NeurIPS 2024 · similarity 0.88)
- The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination (ICML 2025 · similarity 0.87)
- Time Travel in LLMs: Tracing Data Contamination in Large Language Models (ICLR 2024 · similarity 0.87)
- Provable Joint Decontamination for Benchmarking Multiple Large Language Models (NeurIPS 2026 · similarity 0.87)
- How Much Can We Forget about Data Contamination? (ICML 2025 · similarity 0.86)
- How Contaminated Is Your Benchmark? Measuring Dataset Leakage in Large Language Models with Kernel Divergence (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
