AI Paper Atlas

Research map / Large language models / Reasoning and chain of thought

ChainGPT: Dual-Reasoning Model with Recurrent Depth and Multi-Rank State Updates

Yunao Zheng, Xiaojie Wang, Lei Ren, Chen Wei

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Long CoT dynamics and efficiency

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ChainGPT, a dual-reasoning model with recurrent depth and multi-rank state updates to enable deeper latent reasoning than fixed-depth Transformers and chain-of-thought.

Problem

Fixed-depth Transformers struggle with complex reasoning end-to-end, and chain-of-thought relies on natural language generation whose cost grows with sequence length.

Approach

Introduces recurrent depth with multi-rank state updates in a dual-reasoning architecture so that reasoning happens in latent space rather than through lengthy generated text; details beyond the abstract are not available.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
train ChainGPT instead of Qwen2.5 at same parameter count0.5B model · lm-eval six-task averageaverage accuracy: 0.4317 -> 0.4575Improved
train ChainGPT instead of Qwen2.5 at same parameter count1.5B model · lm-eval six-task averageaverage accuracy: 0.5059 -> 0.5291Improved
ChainGPT versus Qwen2.5-0.5B on GLUE and SciQ0.5B model · GLUE, SciQaccuracy: GLUE 0.4664 -> 0.4679; SciQ 0.5290 -> 0.5330No clear effect
add LoRA sub-step (RWKV-Product) to RWKV-7 (ablation)RWKV-7 · modded-nanogpt-rwkv validation lossvalidation loss: 3.2715 -> 3.1901Improved
increase sub-step count M (180M model) (ablation)RWKV-Product · FineWeb validationvalidation lossImproved
decouple state propagation paths instead of coupled (ablation)RWKV-Product · training validationvalidation lossImproved
add State-Guided Sparse Attention to the RWKV-7 state layer (ablation)RWKV-7 · MQAR (2048, 256)accuracy: 72.93% -> >99%Improved
add sparse global anchors (G=32/64/128) to window attention (ablation)SGSA · PG-19 perplexity, 16kperplexity: 16.12 -> 16.08Improved
vary global anchor stride G among 32, 64 and 128 (ablation)SGSA · PG-19 perplexityperplexity: 16.09 (G=32), 16.08 (G=64), 16.08 (G=128) at 16kNo clear effect
increase recurrence iterations from x1 to x16 (ablation)recurrent depth core · validation perplexityperplexityMixed
add recurrence (Loop) to the base architectureQwen3 · GOAT arithmeticaccuracy (%): 33.82 -> 50.54Improved
add recurrence (Loop) to the base architectureRWKV-7 · GOAT arithmeticaccuracy (%): 23.69 -> 24.81No clear effect

Experimental setup

Builds on: chain-of-thought

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy