Research map / Large language models / Reasoning and chain of thought

Think-While-Generating: On-the-Fly Reasoning for Personalized Long-Form Generation

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FlyThinker, a think-while-generating framework for personalized long-form generation in which a separate latent Reasoner runs in parallel with a Generator, with the Reasoner conditioned only on previous response tokens so training and inference stay parallel.

Problem

Think-then-generate personalization uses a static one-shot reasoning trace that must cover the whole long response, which is hard to learn and cannot adapt as content evolves; existing latent/CoT reasoning also serializes reasoning and generation, slowing training and inference.

Approach

A separate Reasoner LLM emits a latent token per position conditioned only on query, history and previously generated response tokens (not its own prior reasoning), so all reasoning tokens can be computed in one teacher-forced forward pass; the latent is added to the Generator's token embeddings with coefficient lambda, and both run staggered in parallel at inference.

Key results

• On Product Review, FlyThinker reaches ROUGE-1 0.3663 and BLEU 4.36 versus 0.3551 and 3.91 for SFT with Qwen2.5-3B-Instruct.
• On Abstract Generation, FlyThinker reaches BLEU 6.34 versus 5.85 for CoT and 5.23 for Coconut.
• Training overhead stays close to SFT while CoT and Coconut are substantially slower; a 0.5B Reasoner with a 7B Generator still improves over SFT on most metrics.

Surprising findings

• Reducing the Reasoner from 3B to 1.5B keeps performance nearly unchanged, and a 0.5B Reasoner with a 7B Generator still helps.
• Input-only latent reasoning augmentation yields higher METEOR but lower ROUGE and BLEU, indicating diversity-alignment tradeoffs.

Limitations

• Gains on ROUGE-L and some metrics are small or mixed; SFT is sometimes stronger on ROUGE-L and BERTScore for some settings.
• Inference requires running two models and extra memory, and computational cost is not reduced.
• Evaluation relies on lexical overlap metrics and a single LLM judge, with no human evaluation.

Experimental setup

Datasets: LongLaMP Product Review LongLaMP Abstract Generation LongLaMP Topic Writing

Benchmarks: LongLaMP Product Review LongLaMP Abstract Generation LongLaMP Topic Writing

Metrics: ROUGE-1 ROUGE-L BLEU METEOR BERTScore

Baselines: Non-pers RAG CoS SFT LLM-TRSR NextQuill CoT Coconut

Builds on: Coconut LongLaMP SFT NextQuill LLM-TRSR

Closest papers by meaning

• Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models (NeurIPS 2025 · similarity 0.88)
• Pruning Long Chain-of-Thought of Large Reasoning Models via Small-Scale Preference Optimization (ICLR 2026 · similarity 0.87)
• Reasoning via Test-Time Instance-Level Policy Gradient in Latent Space (NeurIPS 2026 · similarity 0.87)
• Thinking on the Fly: Test-Time Reasoning Enhancement via Latent Thought Policy Optimization (ICLR 2026 · similarity 0.87)
• Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback (ICML 2025 · similarity 0.87)
• Rethinking Personalized Generation: Test-time Alignment via Factorized Ranking Models (NeurIPS 2026 · similarity 0.87)