Research map / Large language models / Language, safety and interpretability
Reflection-Window Decoding: Text Generation with Selective Refinement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes reflection-window decoding, which uses a sliding window with an entropy-based pausing criterion to trigger beam-search regeneration of recent tokens, interleaving refinement and generation during autoregressive decoding.
Problem
Autoregressive decoding is suboptimal for joint sequence probability because it lacks a built-in mechanism to refine or correct already-generated tokens, so greedy decoding can miss the globally optimal same-length response.
Approach
A sliding reflection window over the generated text with fast and slow pointers; when the conditional entropy of recent tokens exceeds a threshold, the window is regenerated with fixed-length beam search, so refinement and generation alternate during decoding, with a theoretical argument that high uncertainty signals sub-optimal history.
Key results
- On MMLU with Phi3-Medium, reflection-window (greedy) reaches 78.15% macro average vs 78.14% greedy and 76.44% beam search.
- Reflection-window beats Top-k/Top-p sampling on MMLU across four models, with average gains from 0.88 (Mistral-Nemo) to 2.76 (Phi3-Medium) points.
- On MT-Bench with Llama3.1-8B, reflection-window wins against greedy in 66.67% of GPT-4o-judged cases versus 46.3% for beam search.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use reflection-window selective refinement instead of greedy decoding | Phi3-Medium · MMLU macro average (four categories) | accuracy: 78.14% -> 78.15% | Helped |
| Use reflection-window selective refinement instead of beam search | Phi3-Medium · MMLU macro average (four categories) | accuracy: 76.44% -> 78.15% | Helped |
| Use reflection-window decoding instead of Top-k/Top-p sampling | Llama3.1-8B, Phi3-Medium, Qwen2.5-14B, and Mistral-Nemo · MMLU average | accuracy: 66.46% -> 67.21%; 71.97% -> 74.73%; 77.84% -> 78.48%; 59.83% -> 60.71% | Helped |
| Use reflection-window decoding instead of beam search | Llama3.1-8B · MT-Bench | win rate against Greedy Decoding: 46.3% -> 66.67% | Helped |
| Set window size d=3 instead of d=2 or d=4 (ablation) | Reflection-Window on Mistral-Nemo · MT-Bench | overall mean: 7.82 -> 7.93; 7.84 -> 7.93 | Mixed |
| Set window size d=5 instead of d=2, d=3, or d=4 (ablation) | Reflection-Window on Llama3.1-8B · MT-Bench | overall mean: 7.69 -> 7.97; 7.93 -> 7.97; 7.89 -> 7.97 | Helped |
| Set entropy threshold σ=0.25 instead of σ=0.1, σ=0.5, σ=0.75, or σ=1.0 (ablation) | Reflection-Window on Qwen2.5-7B · MMLU Social Sciences | accuracy: 79.40 -> 80.31; 79.82 -> 80.31; 79.88 -> 80.31; 79.69 -> 80.31 | Mixed |
| Set window size d=6 instead of d=2, d=3, d=4, or d=5 (ablation) | Reflection-Window on Qwen2.5-7B · MMLU Social Sciences | accuracy: 79.66 -> 79.88; 79.75 -> 79.88; 79.82 -> 79.88; 79.62 -> 79.88 | Mixed |
| Increase window size from d=2 to d=4 (ablation) | Reflection-Window on Llama3.1-8B · MMLU college-level subjects | average regeneration ratio: 9.60% -> 3.70% | Helped |
Surprising findings
- Vanilla beam search underperforms greedy decoding on MMLU with Phi3-Medium (76.44% vs 78.14%) and on MT-Bench win rate (46.3%).
Limitations
- Theoretical result is only a necessary condition for deviation and relies on an oracle LLM assumption not verified in practice.
- Sanity check uses beam search (width 10) as a proxy for global optimum, which is only an approximation.
- Gains on MMLU are small (e.g., 0.01 to 0.03 points in some settings) and hyperparameter sensitivity is modest in scope.
Experimental setup
Datasets: MMLU MT-Bench
Benchmarks: MMLU MT-Bench
Metrics: macro averaged accuracy win rate GPT-4o judge score regeneration ratio
Baselines: greedy decoding beam search Top-k/Top-p sampling
Builds on: beam search Top-k sampling Top-p sampling speculative decoding
Closest papers by meaning
- Improving Open-Ended Text Generation via Adaptive Decoding (ICML 2024 · similarity 0.90)
- When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models (ICML 2024 · similarity 0.88)
- The Hyperfitting Phenomenon: Sharpening and Stabilizing LLMs for Open-Ended Text Generation (ICLR 2025 · similarity 0.88)
- Multi-Branch Self-Drafting for LLM Inference Acceleration (AAAI 2025 · similarity 0.88)
- Beyond Autoregression: Fast LLMs via Self-Distillation Through Time (ICLR 2025 · similarity 0.88)
- Accelerating Diffusion LLMs via Adaptive Parallel Decoding (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.