Research map / Large language models / Language, safety and interpretability

Orthrus: Memory-Efficient Parallel Token Generation via Dual-View Diffusion

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Parallel and early-exit decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Orthrus adds a lightweight trainable parallel diffusion attention head to a frozen Qwen3 autoregressive model, sharing its KV cache, and uses intra-model greedy consensus verification to get lossless parallel decoding with up to ~7.8x speedup in tokens-per-forward-pass.

Problem

Autoregressive LLM decoding is sequential and memory-bandwidth bound, while diffusion language models that parallelize generation lose accuracy or require heavy training to match AR models.

Approach

Freeze the AR backbone for prefill and KV cache construction, add a trainable diffusion attention path per layer that predicts a block of K tokens in one forward pass over the same KV cache, distill the AR teacher's distribution into it, and verify the drafted block with the frozen AR head so output matches greedy AR decoding exactly.

Key results

• Orthrus-Qwen3-8B reaches average TPF 5.39 and 5.36x speedup over Qwen3-8B across eight tasks under greedy decoding, with peak 7.83x on Pseudo2code.
• On MATH-500, Orthrus-Qwen3-8B matches the base model's accuracy (86.2) with TPF 6.35 (about 6x speedup), while Fast-dLLM-v2 scores 61.5.
• Orthrus trains only about 16% of parameters on ~0.96B tokens in under 24 hours on a single 8xH200 node.

Surprising findings

• Single-step parallel block projection outperforms a 2-step iterative denoising variant (TPF 6.35 vs 3.53).
• Soft KL distillation from the AR teacher gives higher TPF than hard cross-entropy labels (6.35 vs 5.86) at identical accuracy.

Limitations

• Generation capability is upper-bounded by the frozen AR base model, inheriting its biases and errors.
• Speedup is measured in tokens per forward pass rather than wall-clock in most tables, and hardware-agnostic metrics may overstate practical gains.
• Sampling with temperature T>0 relies on rejection sampling whose details are only briefly described.

Experimental setup

Datasets: Nemotron-Post-Training-Dataset-v2

Benchmarks: GSM8K MATH-500 AIME24 AIME25 HumanEval MBPP Pseudo2code LiveCodeBench-v5

Metrics: tokens per forward pass (TPF) speedup accuracy average acceptance length throughput (tokens/s) peak memory KV cache overhead

Baselines: EAGLE-3 DFlash Fast-dLLM-v2 SDAR Dream-7B LLaDA-1.5 Mercury Coder Small Gemini Diffusion Qwen3-8B autoregressive decoding

Builds on: Fast-dLLM-v2 EAGLE-3 FlexAttention FlashAttention

Closest papers by meaning

• Polestar: Drift-Aware Cache Calibration and Token Commitment for Efficient Inference of Diffusion LLMs (NeurIPS 2026 · similarity 0.91)
• Accelerating Diffusion LLMs via Adaptive Parallel Decoding (NeurIPS 2025 · similarity 0.90)
• Mean-Field Parallel Decoding for Discrete Diffusion Language Models (NeurIPS 2026 · similarity 0.90)
• MARS: Enabling Autoregressive Models Multi-Token Generation (NeurIPS 2026 · similarity 0.90)
• Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning (NeurIPS 2025 · similarity 0.89)
• Free Draft-and-Verification: Toward Lossless Parallel Decoding for Diffusion Large Language Models (NeurIPS 2026 · similarity 0.89)