Research map / Large language models / Reasoning and chain of thought

From Imitation to Discrimination: Toward a Generalized Curriculum Advantage Mechanism Enhancing Cross-Domain Reasoning Tasks

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CAPO, a two-phase advantage-based curriculum that first trains on positive-advantage samples only, then adds negative ones, improving GRPO/PPO/RLOO/Reinforce++ on math reasoning and GUI tasks.

Problem

Standard advantage-based RL for LLM reasoning mixes positive and negative advantage samples from the start, which can give ambiguous guidance and limit gains, especially early in training.

Approach

Use the sign of the advantage as a curriculum signal: an imitation phase trains only on positive-advantage samples (with KL regularization), then a discrimination phase switches on negative-advantage samples at a fixed switch point (e.g., 10-30% of steps). The method is plug-in compatible with GRPO, PPO, RLOO and Reinforce++.

Key results

• On Qwen2.5-Math-7B, CAPO improves GRPO average from 48.9 to 52.8 (+3.9) and yields +1.7 to +3.2 across the other three algorithms.
• On Qwen2.5-VL-3B GUI planning, CAPO raises the overall score from 70.79 to 74.60 (+3.81) over GRPO.
• On OOD ARC-C and GPQA-Diamond, CAPO averages 52.8, +3.8 over GRPO according to the text (the appendix claims +6.5%).

Surprising findings

• Gradually introducing negative signals was reported to underperform a simple hard switch point.
• On AndroidControl-Low SR, CAPO is 2.46 points lower than GRPO despite gains elsewhere.

Limitations

• Theoretical justification is heuristic: the variance-bias argument and convergence proposition rely on loose assumptions and the appendix proof is informal.
• Switch point is a hyperparameter selected via sensitivity sweeps (10-40%), and the hard switch is chosen pragmatically.
• Evaluation of math uses a 1.5B/7B Qwen2.5-Math base with small benchmark sets (e.g., AIME24 with 30 problems), so differences of a few points may be noisy.

Experimental setup

Datasets: MATH (Level 3-5, 5.5K subset) GUI-R1-3K OS-Atlas

Benchmarks: AIME 2024 AMC MATH500 GSM8K Minerva OlympiadBench CollegeMath ARC-C GPQA-Diamond GUI-Act-Web OmniAct-Web AndroidControl-Low

Metrics: accuracy (pass@1, greedy) Type/GR/SR accuracy for GUI

Baselines: GRPO PPO RLOO Reinforce++ ADARFT GRPO(+SC) CoT Os-Atlas-4B QwenVL2.5-3B

Builds on: GRPO PPO RLOO REINFORCE++ GUI-R1

Closest papers by meaning

• CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMs (ICLR 2026 · similarity 0.88)
• AdaCuRL: Adaptive Curriculum Reinforcement Learning with Invalid Sample Mitigation and Historical Revisiting (AAAI 2026 · similarity 0.87)
• Beyond English-Centric Training: How Reinforcement Learning Improves Cross-Lingual Reasoning in LLMs (ICLR 2026 · similarity 0.86)
• SATURN: SAT-based Reinforcement Learning to Unleash LLMs Reasoning (NeurIPS 2025 · similarity 0.86)
• DAPO : Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage-Based Policy Optimization (NeurIPS 2025 · similarity 0.86)
• Probing to Refine: Reinforcement Distillation of LLM Reasoners via Explanatory Inversion (ICLR 2026 · similarity 0.86)