Research map / Large language models / Reasoning and chain of thought
ChainGPT: Dual-Reasoning Model with Recurrent Depth and Multi-Rank State Updates
Yunao Zheng, Xiaojie Wang, Lei Ren, Chen Wei
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ChainGPT, a dual-reasoning model with recurrent depth and multi-rank state updates to enable deeper latent reasoning than fixed-depth Transformers and chain-of-thought.
Problem
Fixed-depth Transformers struggle with complex reasoning end-to-end, and chain-of-thought relies on natural language generation whose cost grows with sequence length.
Approach
Introduces recurrent depth with multi-rank state updates in a dual-reasoning architecture so that reasoning happens in latent space rather than through lengthy generated text; details beyond the abstract are not available.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train ChainGPT instead of Qwen2.5 at same parameter count | 0.5B model · lm-eval six-task average | average accuracy: 0.4317 -> 0.4575 | Improved |
| train ChainGPT instead of Qwen2.5 at same parameter count | 1.5B model · lm-eval six-task average | average accuracy: 0.5059 -> 0.5291 | Improved |
| ChainGPT versus Qwen2.5-0.5B on GLUE and SciQ | 0.5B model · GLUE, SciQ | accuracy: GLUE 0.4664 -> 0.4679; SciQ 0.5290 -> 0.5330 | No clear effect |
| add LoRA sub-step (RWKV-Product) to RWKV-7 (ablation) | RWKV-7 · modded-nanogpt-rwkv validation loss | validation loss: 3.2715 -> 3.1901 | Improved |
| increase sub-step count M (180M model) (ablation) | RWKV-Product · FineWeb validation | validation loss | Improved |
| decouple state propagation paths instead of coupled (ablation) | RWKV-Product · training validation | validation loss | Improved |
| add State-Guided Sparse Attention to the RWKV-7 state layer (ablation) | RWKV-7 · MQAR (2048, 256) | accuracy: 72.93% -> >99% | Improved |
| add sparse global anchors (G=32/64/128) to window attention (ablation) | SGSA · PG-19 perplexity, 16k | perplexity: 16.12 -> 16.08 | Improved |
| vary global anchor stride G among 32, 64 and 128 (ablation) | SGSA · PG-19 perplexity | perplexity: 16.09 (G=32), 16.08 (G=64), 16.08 (G=128) at 16k | No clear effect |
| increase recurrence iterations from x1 to x16 (ablation) | recurrent depth core · validation perplexity | perplexity | Mixed |
| add recurrence (Loop) to the base architecture | Qwen3 · GOAT arithmetic | accuracy (%): 33.82 -> 50.54 | Improved |
| add recurrence (Loop) to the base architecture | RWKV-7 · GOAT arithmetic | accuracy (%): 23.69 -> 24.81 | No clear effect |
Experimental setup
Builds on: chain-of-thought
Closest papers by meaning
- Depth-Recurrent Attention Mixtures: Giving Latent Reasoning the Attention it Deserves (NeurIPS 2026 · similarity 0.90)
- Markovian Transformers for Informative Language Modeling (ICLR 2026 · similarity 0.89)
- Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach (NeurIPS 2025 · similarity 0.89)
- SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought (NeurIPS 2025 · similarity 0.89)
- ReaGEN: Adaptive Generation of Structured Chains-of-Thought for Efficient Multimodal Reasoning (CVPR 2026 · similarity 0.89)
- Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
