AI Paper Atlas

Research map / Large language models / Reasoning and chain of thought

MoDr: Mixture-of-Depth-Recurrent Transformers for Test-Time Reasoning

Xiaojing Zhang, Haifeng Wu, Gang He, Jiyang Shen, Bochen Lyu, Zhanxing Zhu

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MoDr, a mixture-of-depth-recurrent Transformer that builds on depth-recurrent reasoning (Huginn) to improve test-time reasoning.

Problem

Depth-recurrent Transformers like Huginn-3.5B increase per-token computation by reusing a latent recurrent block, but their gains with increasing recurrence are the motivating concern; the abstract is truncated so the specific limitation addressed is only partially stated.

Approach

Mixes recurrent depths in a mixture-of-depth-recurrent architecture to enable test-time reasoning in latent space as an alternative to chain-of-thought (details beyond the truncated abstract are not stated).

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace single recurrent module with 4 LoRA branches plus hard routerHuginn-3.5B · 6 math benchmarks (average)accuracy: 62.02 -> 69.22Improved
compare with LoRA fine-tuned single-branch Huginn-SFTMoDr · 6 math benchmarks (average)accuracy: 66.74 -> 69.22Improved
disable the router and select branches randomly at inference (ablation)MoDr · 6 math benchmarks (average)accuracy: 69.22 -> 66.41Hurt
add dynamic router to 4-branch model instead of random selection in training (ablation)4-branch Huginn · GSM8Kaccuracy: 50.72 -> 49.89No clear effect
use dynamic routing instead of single branches (ablation)MoDr · 6 math benchmarks (average)accuracy: 67.94 -> 69.22 (best single branch)Improved
add auxiliary-loss-free load balancing (eta 0.001) (ablation)MoDr · 6 math benchmarks (average)accuracy: 68.66 -> 69.22Improved
increase number of recurrent branches beyond 4 (ablation)MoDr · 6 math benchmarks (average)accuracyNo clear effect
use Top-2 instead of Top-1 routing (ablation)MoDr · 6 math benchmarks (average)accuracy: 69.22 -> 71.60; AQuA 33.07 -> 28.35Mixed
use full fine-tuning instead of LoRA (ablation)Huginn · 6 math benchmarks (average)accuracy: 69.22 -> 42.56Hurt
add few-shot CoT prompting to HuginnHuginn · 6 math benchmarksaccuracyNo clear effect
use MoDr instead of Huginn-SFT on commonsenseMoDr · HellaSwagaccuracy: 80.98 -> 80.15Hurt
use MoDr instead of Huginn-SFT on commonsenseMoDr · 6 commonsense benchmarks (average)accuracy: 76.86 -> 78.38Improved

Limitations

Experimental setup

Baselines: Huginn-3.5B

Builds on: Huginn

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy