Research map / Large language models / Reasoning and chain of thought

SCOPE: Self-Play via Co-Evolving Policies for Open-Ended Tasks

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SCOPE, a data-free self-play framework where a Challenger generates document-grounded open-ended tasks and a Solver answers them via multi-turn retrieval, with a frozen self-judge writing rubrics and grading responses.

Problem

Self-play for LLMs requires rule-checkable answers, so open-ended tasks depend on curated prompts or frontier-model judges; the paper asks whether data-free self-play can work for open-ended tasks.

Approach

Co-evolve a Challenger (task generator from Wikipedia documents) and a Solver (multi-turn retrieval answerer) with rewards from a frozen base-model Judge that writes task-specific rubrics from the source document and grades responses; the Challenger is rewarded for tasks at mid-difficulty (rubric score near 0.5), and quality gates and a cosine length penalty block reward hacking.

Key results

• On open-ended benchmarks SCOPE gains +10.4 on Qwen2.5-7B (24.4 to 34.8), +7.8 on OLMo-3-7B (30.7 to 38.5), and +5.4 on Qwen3-8B (37.7 to 43.1), comparable to GRPO_data (33.4, 39.0, 41.5 averages for Qwen2.5, OLMo-3, Qwen3 respectively; OLMo-3 and Qwen3 GRPO_data are higher on OLMo-3).
• On seven short-form QA benchmarks, SCOPE improves averages by up to +13.8 points and surpasses GRPO_data on all three models (e.g., 61.8 vs 60.7 on Qwen3-8B).
• A frozen Challenger gains only +0.8 from iter-1 to iter-3 versus +3.4 for co-evolution on Qwen3-8B.

Surprising findings

• On OLMo-3-7B, SCOPE is below GRPO_data on short-form QA averages (49.2 vs 48.6 at iter-3 is actually above, but on open-ended average GRPO_data remains higher, 39.0 vs 38.5).
• A 4B rubric generator drops performance by 2.8 points while scaling the grader alone changes it by only 0.7 points, showing rubric quality matters more than grading capacity.

Limitations

• Multi-stage pipeline (Challenger training, rubric generation, filtering) costs more compute than GRPO on curated prompts.
• Experiments limited to 7-8B models; gains at larger scales untested.
• Quality gates do not explicitly screen harmful content from generated tasks.

Experimental setup

Datasets: 2018 English Wikipedia DR Tulu curated prompts (for GRPO_data baseline)

Benchmarks: DRB-RACE ResearchRubrics ResearchQA SQA-CS-V2 ResearchPlanGen HealthBench WildBench Arena-Hard-CW NQ TriviaQA PopQA HotpotQA

Metrics: average score (LLM-judged rubric/accuracy)

Baselines: GRPO_data DR Tulu WebExplorer WebThinker-R1 frozen Challenger ablation

Builds on: GRPO Absolute Zero R-Zero RaR DR Tulu RPG

Closest papers by meaning

• Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models (ICML 2024 · similarity 0.87)
• SPELL: Self-Play Reinforcement Learning for Evolving Long-Context Language Models (ICLR 2026 · similarity 0.87)
• Self-Improvement in Language Models: The Sharpening Mechanism (ICLR 2025 · similarity 0.87)
• SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Models (ICLR 2025 · similarity 0.86)
• SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoning (NeurIPS 2025 · similarity 0.86)
• Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing (NeurIPS 2024 · similarity 0.86)