Research map / Large language models / Language, safety and interpretability

Provable Joint Decontamination for Benchmarking Multiple Large Language Models

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Uncertainty quantification for LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Joint Envelope Conformal Selection (JECS), which aggregates per-model conformal p-values by max and recalibrates them with a fitted right-tail envelope before adaptive Storey-BH, to select one shared decontaminated benchmark with provable global contamination rate control across multiple audited LLMs.

Problem

Selecting a single benchmark subset that is free of training-data contamination across multiple audited LLMs, with provable control of the expected fraction of contaminated items; per-model conformal selections are model-specific and naive union/intersection fail to control error.

Approach

Test the joint purity null via the per-item maximum of model-wise conformal p-values, then reconstruct a conservative envelope of the max-p null CDF from right-tail observations and apply adaptive Storey-BH to envelope-transformed values, recovering power lost to super-uniformity of the max statistic.

Key results

• On ArXivTection with Pythia-6.9B and Min-K%++ at alpha=0.1, JECS improves power from 0.094 (JMCS) to 0.447.
• On the synthetic setup at alpha=0.1, JECS attains GCR 0.038 while union and intersection reach 0.763 and 0.366, violating the target.
• Under LLaMA-7B at alpha=0.1 with Min-K%, JECS raises power from 0.055 to 0.715 over JMCS at K=16 on ArXivTection.

Surprising findings

• Naive union and intersection of per-model conformal selections severely violate the GCR target, with union reaching about 0.76 contamination at all alpha levels.

Limitations

• Requires a shared calibration set of known members of every audited model whose distribution closely matches the candidate pool; distribution shift is not handled.
• GCR control is asymptotic (limsup as n grows) for the main theorem, relying on assumptions such as convexity of the null max-p CDF and right-tail distinguishability.
• Experiments use controlled fine-tuning with synthetic membership labels rather than real pretraining corpora.

Experimental setup

Datasets: WikiMIA ArXivTection MIMIR HackerNews MIMIR DM-Math MIMIR GitHub MIMIR Pile-CC MIMIR PubMed Central

Benchmarks: WikiMIA ArXivTection MIMIR subsets (HackerNews, DM-Math, GitHub, Pile-CC, PubMed Central) synthetic setup

Metrics: global contamination proportion (GCP) global contamination rate (GCR) power

Baselines: JMCS (max-p conformal selection) union of per-model selections intersection of per-model selections Min-K%++ Min-K% Perplexity Modified Entropy

Builds on: conformal selection Benjamini-Hochberg procedure Storey's estimator Min-K%++ conformal p-values

Closest papers by meaning

• The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination (ICML 2025 · similarity 0.88)
• ConStat: Performance-Based Contamination Detection in Large Language Models (NeurIPS 2024 · similarity 0.88)
• How Much Can We Forget about Data Contamination? (ICML 2025 · similarity 0.87)
• Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification (ICLR 2025 · similarity 0.87)
• Time Travel in LLMs: Tracing Data Contamination in Large Language Models (ICLR 2024 · similarity 0.86)
• Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination’s Impact on Machine Translation (ICML 2025 · similarity 0.86)