Research map / Large language models / Language, safety and interpretability

XTC: Head-Aware Sampling by Excluding Top Choices

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Parallel and early-exit decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes XTC, a head-aware decoding operator that removes dominant plausible tokens (keeping the weakest eligible one) with probability rho, and evaluates it on creative diversity, quality, and IFEval across Gemma 3, DeepSeek R1 14B, and Llama 3.3 70B.

Problem

Autoregressive decoding samplers (temperature, top-k/top-p, typical, min-p) target the distribution tail or global entropy and overlook the head-ambiguity regime, where several plausible tokens exist but the most generic one dominates, causing homogenized open-ended generation.

Approach

When two or more tokens exceed an absolute probability threshold tau, with probability rho remove all eligible tokens except the least probable one and renormalize; the operator is inactive otherwise and composes with any sampler stack.

Key results

• On creative generation, XTC improves Distinct-2 by 11-15% and reduces repeat trigram rate by 27-47% across four tested models (12B to 70B).
• On Llama 3.3 70B IFEval, XTC Medium shifts prompt-level strict accuracy by -1.7 pp while temperature 1.15 matched on Distinct-2 drops it by -8.8 pp.
• In a blinded AMT study with 150 Master workers, XTC was preferred for creativity in 62.3% of comparisons (p<1e-4) and ties or beats baseline on quality in 84.4% of comparisons.

Surprising findings

• Top-p and typical-p at 0.95 reduce Distinct-2 relative to baseline on the creative suite, suggesting tail truncation can worsen head dominance.
• Temperature matched on Distinct-2 degrades IFEval by 8.8 points while XTC degrades it by only 1.7 points.

Limitations

• XTC can hurt exactness-sensitive tasks such as code, extraction, and formatting; pass rates degrade above rho=0.15 on code.
• The absolute threshold tau is sensitive to model scale, tokenizer, and upstream sampler stack, so optimal settings vary across models.
• Evaluation is restricted to quantized open-weight models in the 12B-70B range and English-only settings.

Experimental setup

Datasets: Creative Writing Bench v3 prompts (EQ-Bench) IFEval

Benchmarks: Creative Writing Bench v3 prompts IFEval code exactness tasks JSON extraction

Metrics: Distinct-2 Self-BLEU-4 repeat trigram rate embedding cosine distance compression ratio IFEval prompt-level strict accuracy pairwise win rate

Baselines: temperature scaling top-p typical-p repetition penalty eta sampling min-p sampling

Builds on: nucleus sampling min-p sampling locally typical sampling temperature scaling

Closest papers by meaning

• Entropy-Gated Latent Recursion (NeurIPS 2026 · similarity 0.89)
• Entropy-informed Decoding: Adaptive Information-Driven Branching (ICML 2026 · similarity 0.89)
• Large Language Models Explore by Latent Distilling (ICML 2026 · similarity 0.88)
• Test-Time Scaling in the Wild: Why Exploitation, Not Exploration, Is the Bottleneck (NeurIPS 2026 · similarity 0.88)
• Latent Refinement Decoding: Enhancing Diffusion Language Models by Refining Belief States (NeurIPS 2026 · similarity 0.88)
• Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment (ICLR 2025 · similarity 0.88)