Research map / Large language models / Reasoning and chain of thought
MoDr: Mixture-of-Depth-Recurrent Transformers for Test-Time Reasoning
Xiaojing Zhang, Haifeng Wu, Gang He, Jiyang Shen, Bochen Lyu, Zhanxing Zhu
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoDr, a mixture-of-depth-recurrent Transformer that builds on depth-recurrent reasoning (Huginn) to improve test-time reasoning.
Problem
Depth-recurrent Transformers like Huginn-3.5B increase per-token computation by reusing a latent recurrent block, but their gains with increasing recurrence are the motivating concern; the abstract is truncated so the specific limitation addressed is only partially stated.
Approach
Mixes recurrent depths in a mixture-of-depth-recurrent architecture to enable test-time reasoning in latent space as an alternative to chain-of-thought (details beyond the truncated abstract are not stated).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace single recurrent module with 4 LoRA branches plus hard router | Huginn-3.5B · 6 math benchmarks (average) | accuracy: 62.02 -> 69.22 | Improved |
| compare with LoRA fine-tuned single-branch Huginn-SFT | MoDr · 6 math benchmarks (average) | accuracy: 66.74 -> 69.22 | Improved |
| disable the router and select branches randomly at inference (ablation) | MoDr · 6 math benchmarks (average) | accuracy: 69.22 -> 66.41 | Hurt |
| add dynamic router to 4-branch model instead of random selection in training (ablation) | 4-branch Huginn · GSM8K | accuracy: 50.72 -> 49.89 | No clear effect |
| use dynamic routing instead of single branches (ablation) | MoDr · 6 math benchmarks (average) | accuracy: 67.94 -> 69.22 (best single branch) | Improved |
| add auxiliary-loss-free load balancing (eta 0.001) (ablation) | MoDr · 6 math benchmarks (average) | accuracy: 68.66 -> 69.22 | Improved |
| increase number of recurrent branches beyond 4 (ablation) | MoDr · 6 math benchmarks (average) | accuracy | No clear effect |
| use Top-2 instead of Top-1 routing (ablation) | MoDr · 6 math benchmarks (average) | accuracy: 69.22 -> 71.60; AQuA 33.07 -> 28.35 | Mixed |
| use full fine-tuning instead of LoRA (ablation) | Huginn · 6 math benchmarks (average) | accuracy: 69.22 -> 42.56 | Hurt |
| add few-shot CoT prompting to Huginn | Huginn · 6 math benchmarks | accuracy | No clear effect |
| use MoDr instead of Huginn-SFT on commonsense | MoDr · HellaSwag | accuracy: 80.98 -> 80.15 | Hurt |
| use MoDr instead of Huginn-SFT on commonsense | MoDr · 6 commonsense benchmarks (average) | accuracy: 76.86 -> 78.38 | Improved |
Limitations
- Full text unavailable; only truncated abstract was provided
Experimental setup
Baselines: Huginn-3.5B
Builds on: Huginn
Closest papers by meaning
- Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach (NeurIPS 2025 · similarity 0.89)
- Depth-Recurrent Attention Mixtures: Giving Latent Reasoning the Attention it Deserves (NeurIPS 2026 · similarity 0.89)
- A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers (NeurIPS 2025 · similarity 0.88)
- LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation (ICLR 2026 · similarity 0.88)
- THE PATH OF LEAST RESISTANCE: GUIDING LLM REASONING TRAJECTORIES WITH PREFIX CONSENSUS (ICLR 2026 · similarity 0.87)
- R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routing (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
