Research map / Large language models / Reasoning and chain of thought

StreamingThinker: Large Language Models Can Think While Reading

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Chain-of-thought training and analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes StreamingThinker, a streaming "think while reading" paradigm for LLMs that interleaves reasoning with incoming input via streaming CoT data, streaming attention masks/position encoding, and parallel KV caches, cutting time-to-answer latency by over 60% with comparable accuracy on Qwen3 models.

Problem

Standard LLM reasoning (batch thinking) starts only after the full input is available, causing latency and diluted attention to earlier context in dynamic, streaming scenarios.

Approach

Reasoning unfolds sentence-by-sentence alongside the input stream under a streaming attention mask and independent position IDs, with parallel source/target KV caches enabling concurrent reading and reasoning, followed by a depth-controllable final reasoning step once reading completes.

Key results

• On Qwen3-4B, streaming D3 matches batch accuracy on GSM-Symbolic (0.856 vs 0.855) and ProofWriter (0.813 vs 0.620), with delay reduced from 47.70s to 9.77s on GSM-Symbolic.
• Streaming reduces TTFT from 94.74 to 20.77 tokens on GSM-Symbolic (about 80% reduction) and first-token latency from 28.00s to 6.23s in the efficiency analysis (~4.5x).
• Streaming D3 on Qwen3-4B achieves higher accuracy than batch-original on ProofWriter (0.813 vs 0.620) and LogicNLI-style tasks, while using fewer tokens in batch-S settings.

Surprising findings

• Streaming D1 (direct answer) collapses accuracy on several tasks (e.g., 0.1795 on MetaMathQA for Qwen3-1.7B), showing reflection/global depth is needed to approach batch performance.
• The question-first ordering yields higher token usage at D1 but smaller incremental growth at deeper depths, while context-first is more conservative and inefficient on sparse QA tasks.

Limitations

• Evaluated only on deterministic math, logic, and context-QA tasks; broader streaming scenarios are future work.
• Training data depends on large teacher models (Qwen3-32B and Qwen3-235B) for generation and reconstruction.
• Context-first ordering leads to longer generation on sparse QA tasks due to conservative reasoning.

Experimental setup

Datasets: GSM-Symbolic MetaMathQA LogicNLI ProofWriter PubMedQA HotpotQA

Benchmarks: GSM-Symbolic MetaMathQA ProofWriter LogicNLI HotPotQA PubMedQA

Metrics: Pass@1 accuracy token usage TTFT (token-to-first-token) time delay to first answer token

Baselines: Batch, Original Batch, SFT (CoT distillation) Interleaved thinking

Builds on: chain-of-thought RoPE interleaved thinking wait-k policy

Closest papers by meaning

• Think-as-You-See: Streaming Chain-of-Thought Reasoning for Large Vision-Language Models (CVPR 2026 · similarity 0.92)
• Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens (ICML 2026 · similarity 0.90)
• A State-Transition Framework for Efficient LLM Reasoning (ICLR 2026 · similarity 0.90)
• Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning (NeurIPS 2025 · similarity 0.89)
• Accordion-Thinking: Self-Regulated Step Summaries for Efficient and Readable LLM Reasoning (ICML 2026 · similarity 0.89)
• SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought (NeurIPS 2025 · similarity 0.89)