Research map / Architectures and efficiency / Attention and transformers

Decoder-Hybrid-Decoder Architecture for Efficient Reasoning with Long Generation

NeurIPS 2025 · Architectures and efficiency › Attention and transformers · approach: State space and recurrent sequence models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Gated Memory Unit (GMU) for cross-layer memory sharing and the SambaY decoder-hybrid-decoder architecture, scales it with a μP++ hyperparameter scheme, and builds Phi4-mini-Flash(-Reasoning) for efficient long-generation reasoning.

Problem

Decoding efficiency of hybrid SSM/attention LLMs degrades on long generations because cross-attention layers in decoder-decoder models like YOCO still incur linear KV-cache memory I/O during decoding.

Approach

A Gated Memory Unit reuses the SSM mixed-token representation from a earlier self-decoder layer through cheap element-wise gating, replacing half of YOCO's cross-attention layers; this gives constant-size memory reads during decoding while keeping linear pre-filling.

Key results

• SambaY has irreducible loss C=0.58 under FLOPs scaling, versus a higher value for Samba+YOCO, indicating better scaling potential.
• Phi4-mini-Flash-Reasoning reaches 52.29 AIME24, 33.59 AIME25, 92.45 Math500 and 45.08 GPQA Diamond vs Phi4-mini-Reasoning's 48.13, 31.77, 91.20 and 44.51, without RL.
• Phi4-mini-Flash-Reasoning delivers up to 10x higher decoding throughput on 2K prompts with 32K generation under vLLM, and 4.9x speedup in long-context processing.

Measured effects reported in the paper

Surprising findings

• Transformer++ reaches an irreducible loss nearly identical to hybrid models under data scaling (C=1.82), suggesting it could catch up with more data, though under joint model-and-data scaling it has the highest irreducible loss.
• SambaY with a small SWA of 256 outperforms Samba+YOCO with SWA 1024 on long-context retrieval, and larger SWA does not consistently help Phonebook accuracy.

Limitations

• Differential Attention in vLLM uses a naive four-pass FlashAttention rather than an optimized kernel, leaving throughput potential unrealized.
• Phi4-mini-Flash-Reasoning was trained only with SFT and DPO, without RL, and the optimization setup was not optimal; severe loss divergence required stabilization tricks.
• Retains a full attention layer with linear decoding complexity, so extremely long generation remains costly.

Experimental setup

Datasets: SlimPajama ProLong-64K Phi4-mini data corpus Phi4-mini-Reasoning distillation data

Benchmarks: Phonebook RULER Wikitext LAMBADA ARC-c ARC-e HellaSwag PIQA WinoGrande MMLU MMLU-Pro Arena Hard

Metrics: accuracy perplexity irreducible loss pass@1 throughput (MTPS / tokens per second) ROUGE-L F1 win rate

Baselines: Transformer++ Samba+YOCO SWA+YOCO TransformerLS Phi4-mini Phi4-mini-Reasoning DeepSeek-R1-Distill-Qwen-7B DeepSeek-R1-Distill-Llama-8B OpenThinker-7B Bespoke-Stratos-7B

Builds on: YOCO Samba Mamba Differential Attention Phi4-mini-Reasoning

Closest papers by meaning

• Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding (ICML 2025 · similarity 0.88)
• You Only Cache Once: Decoder-Decoder Architectures for Language Models (NeurIPS 2024 · similarity 0.88)
• Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling (ICLR 2025 · similarity 0.87)
• Reward-Guided Speculative Decoding for Efficient LLM Reasoning (ICML 2025 · similarity 0.87)
• Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling (ICLR 2026 · similarity 0.87)
• AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelism (ICML 2025 · similarity 0.87)