Research map / Large language models / Reasoning and chain of thought

Large Language Models Explore by Latent Distilling

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ESamp, a decoding method that trains a lightweight test-time Distiller to predict last-layer hidden states from first-layer states and uses its prediction error as a novelty signal to reweight token logits for semantic diversity.

Problem

Standard stochastic sampling produces lexically varied but semantically redundant candidates, limiting the diversity needed for test-time scaling (Pass@k) of LLMs.

Approach

Use an online-trained shallow-to-deep hidden-state distiller whose prediction error (RND-style novelty) is projected through the LM head to reweight the base logits as (1+β)logit_ref − β logit_dist, biasing decoding toward under-explored semantic regions; parallel samples share the distiller so they diverge collectively.

Key results

• On AIME25 with Qwen2.5-7B-Instruct, ESamp (β=0.25) reaches 46.7% Pass@64 versus lower values for baselines; on GPT-OSS-20B it reaches Pass@64-level baseline performance with only Pass@8.
• Creative writing on BookCorpus with Qwen2.5-7B-Instruct: ESamp achieves Vendi 1.67 and similarity 0.57 with PPL 3.55, better than vanilla (Vendi 1.62, PPL 4.08).
• Asynchronous implementation on RTX4090 (Qwen3-8B) has 0.3% overhead at B=1,K=1 and 4.25% at B=32,K=16 (1.2% in optimized release per abstract).

Surprising findings

• Magnitude-matched random Gaussian noise in place of the distiller error does not reproduce ESamp's gains, while a vocabulary-space distiller performs far worse than latent-space ESamp.
• On AIME25 with Qwen2.5-7B-Instruct, ESamp reduces Pass@1 (6.0% vs 6.6% vanilla) and Pass@8 for Qwen3-8B is lower than vanilla in multi-seed results.

Limitations

• Gains are concentrated on math tasks and reasoning models; coding and QA gains are mixed.
• Pass@1 decreases in some settings (e.g., Qwen2.5-7B on AIME25).
• Throughput numbers in main paper come from an internal implementation; open-source version differs.

Experimental setup

Datasets: AIME 2024 AIME 2025 GPQA-Diamond LiveCodeBench v5 BookCorpus

Benchmarks: AIME 2024 AIME 2025 GPQA-Diamond LiveCodeBench v5 BookCorpus story continuation

Metrics: Pass@k Vendi score embedding cosine similarity perplexity throughput (tokens/sec)

Baselines: Vanilla temperature sampling Min-p FIRE Tree of Thoughts Contrastive Decoding OverRIDE

Builds on: Random Network Distillation KL-regularized policy optimization Controlled Decoding OverRIDE Min-p sampling

Closest papers by meaning

• Semantic-guided Diverse Decoding for Large Language Model (NeurIPS 2025 · similarity 0.89)
• XTC: Head-Aware Sampling by Excluding Top Choices (NeurIPS 2026 · similarity 0.88)
• Free Lunch for Pass@$k$? Low Cost Diverse Sampling for Diffusion Language Models (NeurIPS 2026 · similarity 0.88)
• Entropy-informed Decoding: Adaptive Information-Driven Branching (ICML 2026 · similarity 0.88)
• Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies (ICML 2025 · similarity 0.87)
• SimpleStrat: Diversifying Language Model Generation with Stratification (NeurIPS 2025 · similarity 0.87)