Research map / Large language models / Reasoning and chain of thought

SPELL: Self-Play Reinforcement Learning for Evolving Long-Context Language Models

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SPELL, a single-model multi-role self-play RL framework (questioner, responder, verifier) that improves long-context reasoning without human labels or rule-verifiable rewards.

Problem

Long-context reasoning in LLMs lags behind other capabilities because human annotations are costly and unreliable for long documents, and long-context answers lack programmatically verifiable rewards.

Approach

One policy alternates as questioner (generates questions with reference answers from raw documents), responder (answers using documents), and verifier (judges semantic equivalence), trained jointly with GRPO; a Gaussian questioner reward centered at 0.5 responder success, a history memory for curriculum, and role-specific dynamic sampling.

Key results

• SPELL improves Qwen2.5-7B, 14B, and 32B base models by 13.9, 14.4, and 9.1 average points at 16K input.
• On Qwen3-30B-A3B-Thinking, SPELL gains 2.0 average points at 16K and 2.3 at 100K over base, while RLVR gives no gain at 16K (+0.0).
• SPELL-trained Qwen3-30B-A3B-Thinking reaches pass@8 of 74.5 at 100K versus 68.1 for RLVR and 66.9 for base, and surpasses gemini-2.5-pro at pass@4.

Surprising findings

• Trained base models often outperform their instruction-tuned counterparts that used large-scale human-annotated data.
• Using a stronger external judge (gpt-oss-120b) gave no overall gain over the rule-based judge with verifier (both 47.2 average).

Limitations

• Experiments limited to 16K input context training; generalization to 128K+ not tested for training.
• No theoretical framework explaining co-evolution dynamics of the three roles.
• Still relies on human intervention for corpus preprocessing and prompt template design.

Experimental setup

Datasets: DocMath Ultra-FineWeb LongBench-V2 Frames HotpotQA 2WikiMultihopQA MuSiQue DocMath-Eval testmini

Benchmarks: DocMath Frames LongBench-V2 HotpotQA 2WikiMultihopQA MuSiQue

Metrics: accuracy (pass@1, average of 8 runs) pass@k cover exact match

Baselines: RLVR with DeepSeek-R1-0528 synthesized data AZR reward mapping R-Zero reward mapping Qwen2.5-7B-Instruct Qwen3-30B-A3B-Thinking gemini-2.5-pro

Builds on: GRPO REINFORCE++ DAPO RLVR R-Zero Absolute Zero Reasoner

Closest papers by meaning

• LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contexts (ICLR 2026 · similarity 0.90)
• Thinker: Learning to Think Fast and Slow (NeurIPS 2025 · similarity 0.90)
• TTRL: Test-Time Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
• ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models (NeurIPS 2025 · similarity 0.90)
• SeRL: Self-play Reinforcement Learning for Large Language Models with Limited Data (NeurIPS 2025 · similarity 0.89)
• LongRLVR: Long-Context Reinforcement Learning Requires Verifiable Context Rewards (ICLR 2026 · similarity 0.89)