Research map / Image generation / Fast diffusion sampling

Free Lunch for Pass@$k$? Low Cost Diverse Sampling for Diffusion Language Models

NeurIPS 2026 · Image generation › Fast diffusion sampling · approach: Fast diffusion ODE solvers · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ODD, a training-free inference-time intervention that sequentially repels each diffusion LM sample's max-pooled confidence feature from the span of earlier samples in a batch, improving Pass@k on HumanEval and GSM8K with LLaDA-8B-Instruct.

Problem

Independent samples from diffusion language models collapse into similar modes, wasting compute for Pass@k on reasoning and code tasks, and prior inference-time diversity methods are mostly AR-specific beam variants with high overhead.

Approach

Extract per-sample vocabulary-sized max-pooled confidence features during denoising, and sequentially push each sample's logits to maximize the norm of its residual orthogonal to a Gram-Schmidt basis of previous samples, weighted by a quality score, with stop-gradients on the basis; the update is batch-size invariant.

Key results

• On HumanEval at temperature 1.0, ODD (alpha=8) raises Pass@16 from 42.7 (baseline) to 51.3 with LLaDA-8B-Instruct.
• On GSM8K at temperature 0.0, ODD (alpha=128) raises Pass@16 from 61.0 to 79.0.
• Wall-time overhead is reported as +5.8% for GSM8K and +3.9% for HumanEval at batch size 16.

Surprising findings

• At high temperature (1.5 and above), ODD reduces batch diversity relative to baseline, acting as a coherence filter.
• Global DPP joint optimisation (DiverseFlow) underperforms the greedy sequential ODD objective on Pass@k in this setting.

Limitations

• Evaluated only on LLaDA-8B-Instruct with 4-bit quantisation and on two benchmarks, with GSM8K restricted to 200 problems.
• Large step sizes hurt HumanEval performance and high temperature (2.0) degrades results.
• On GSM8K, ODD slightly reduces cumulative problem coverage relative to baseline and trades Pass@1 for Pass@16.

Experimental setup

Datasets: GSM8K HumanEval

Benchmarks: HumanEval GSM8K

Metrics: Pass@16 Pass@1 cumulative coverage wall-time overhead batch diversity (1 - cosine similarity)

Baselines: LLaDA standard sampling LLaDA + DPP (DiverseFlow)

Builds on: DiverseFlow LLaDA determinantal point process

Closest papers by meaning

• Breaking AR’s Sampling Bottleneck: Provable Acceleration via Diffusion Language Models (NeurIPS 2025 · similarity 0.90)
• Theoretical Benefit and Limitation of Diffusion Language Model (NeurIPS 2025 · similarity 0.90)
• SemanticDLM+: Improving Diffusion LLMs through Bias-variance Trade-off in Transition Kernel Design (NeurIPS 2026 · similarity 0.90)
• Denoising Time Matters: Diverse Generation in Diffusion Language Models (NeurIPS 2026 · similarity 0.89)
• Scaling Diffusion Language Models via Adaptation from Autoregressive Models (ICLR 2025 · similarity 0.89)
• Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling (ICLR 2025 · similarity 0.89)