Research map / Architectures and efficiency / Attention and transformers

Depth-Recurrent Attention Mixtures: Giving Latent Reasoning the Attention it Deserves

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Attention head mechanisms · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Dreamer, a single-layer depth-recurrent attention mixture combining sequence, depth, and sparse expert attention, and shows FLOP-, parameter-, and memory-matched gains on math reasoning over a layered MoE baseline.

Problem

Depth-recurrent models for latent reasoning are hard to scale because a dense recurrent layer is costly and constant hidden-size restricts many-step latent reasoning; prior work also lacks resource-matched ablations.

Approach

A modular depth-recurrent layer that mixes sequence attention, depth attention (attention over previous depths of the same token), and sparse expert attention (MoE with query/key routing), with resource-matched comparisons against layered and non-attention-along-depth variants.

Key results

• Depth-32 DR+DA reaches 51.4 average accuracy vs 39.5 for depth-32 layered baseline, with 3.5x data efficiency.
• Depth-16 DR+DA outperforms depth-32 layered baseline on all reasoning benchmarks at roughly half the parameters, FLOPs, and memory.
• DR uses 2-11x more distinct experts per depth than the layered baseline, with global expert balancing Gini of about 0.075.

Surprising findings

• Depth-16 DR without depth attention outperforms DR+DA on half the reasoning benchmarks, suggesting depth attention helps mainly at greater depth.
• Depth-recurrence without depth attention gives a smaller gain than DR+DA at depth 32, and gains over layered baseline grow with depth for DR+DA.

Limitations

• Results only on math-focused data and benchmarks at ~1B and ~2B scale, trained on ca. 100B tokens.
• Depth attention adds a memory movement overhead; no dynamic-depth or depth-generalization evaluation.
• Scaling laws are not established; the paper defers rigorous scaling analysis to future work.

Experimental setup

Datasets: EleutherAI/hendrycks_math bethgelab/CuratedThoughts microsoft/orca-math-word-problems-200k TIGER-Lab/MathInstruct openai/gsm8k XinyaoHu/AMPS_khan open-thoughts/OpenThoughts2-1M AI-MO/NuminaMath-CoT facebook/natural_reasoning nvidia/OpenMathInstruct-2 nvidia/OpenMathReasoning zwhe99/DeepMath-103K

Benchmarks: GSM8K MATH MMLU (math subset) MathQA Maths-College perplexity

Metrics: accuracy (0-shot) validation perplexity data efficiency factor expert usage Gini coefficient

Baselines: LA (layered MoE baseline, depth 16 and 32) DR (depth recurrence without depth attention)

Builds on: DeepSeek-V3 auxiliary-loss-free balancing Transformer self-attention grouped-query attention RoPE SwiGLU

Closest papers by meaning

• Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach (NeurIPS 2025 · similarity 0.88)
• RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling (NeurIPS 2025 · similarity 0.88)
• Visual Enhanced Depth Scaling for Multimodal Latent Reasoning (NeurIPS 2026 · similarity 0.87)
• Dual-Latent Memory Routing for Vision-Language Reasoning (ICML 2026 · similarity 0.87)
• Efficient Post-Training Refinement of Latent Reasoning in Large Language Models (AAAI 2026 · similarity 0.87)
• Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation (NeurIPS 2025 · similarity 0.87)