Research map / Large language models / Reasoning and chain of thought

Less Diverse, Less Safe: The Indirect But Pervasive Risk of Test-Time Scaling in Large Language Models

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that constraining candidate diversity makes test-time scaling (Best-of-N and MCTS) more likely to select unsafe outputs, via a genetic-algorithm stress test (RefDiv) that transfers to closed-source models and bypasses guardrail classifiers.

Problem

Test-time scaling is assumed to be robust when candidate pools are diverse, but its safety under reduced candidate diversity is unknown; the paper asks whether adversarial prompts can collapse diversity and make TTS output harmful responses.

Approach

RefDiv is a population-based genetic algorithm (extending AutoDAN) whose fitness combines a reference-guided affirmative-token entropy term with intrinsic Shannon entropy of TTS candidate sets, with a time-varying weight that shifts from reference steering to pure diversity minimization.

Key results

• On Best-of-N (N=8, PairRM), RefDiv reaches 0.465 ASR on Llama3.1-8B vs 0.368 for AutoDAN and 0.926 on Gemma3-27B vs 0.749 for AutoDAN.
• On MCTS, RefDiv achieves 0.967 ASR on Llama3.1-8B vs 0.831 for AutoDAN and 0.989 on Gemma3-27B vs 0.904 for AutoDAN.
• Guardrail classifiers are bypassed by RefDiv prompts from Llama3.1-8B with average ASR of about 82% on Best-of-N.

Surprising findings

• Successful attacks require only a small entropy drop (about 2-5%) relative to initial entropy.
• Increasing the number of Best-of-N candidates does not effectively reduce attack success.

Limitations

• Transfer to o3-mini is relatively low (highest ASR 0.34), and results are limited to a single benchmark (AdvBench).
• Attack success is judged by absence of affirmative/refusal strings following AutoDAN protocol rather than a stronger safety judge.
• Mitigation analysis is limited: perplexity filtering and a safety-aware reward model only partially reduce effectiveness; no defense is proposed.

Experimental setup

Datasets: AdvBench

Benchmarks: AdvBench

Metrics: attack success rate Shannon entropy Pearson correlation between entropy and ASR

Baselines: GCG AutoDAN AutoDAN-Turbo

Builds on: AutoDAN Best-of-N sampling Monte Carlo Tree Search GCG AutoDAN-Turbo

Closest papers by meaning

• Test-Time Scaling in the Wild: Why Exploitation, Not Exploration, Is the Bottleneck (NeurIPS 2026 · similarity 0.89)
• Understanding the Role of Training Data in Test-Time Scaling (ICLR 2026 · similarity 0.88)
• Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.88)
• SimpleStrat: Diversifying Language Model Generation with Stratification (NeurIPS 2025 · similarity 0.88)
• On Evaluating the Durability of Safeguards for Open-Weight LLMs (ICLR 2025 · similarity 0.87)
• Test-Time Learning for Large Language Models (ICML 2025 · similarity 0.87)