Research map / Large language models / Reasoning and chain of thought

Rectify Evaluation Preference: Improving LLMs’ Critique on Math Reasoning via Perplexity-aware Reinforcement Learning

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes perplexity-aware GRPO, which rescales advantages by solution perplexity and uses class-level loss aggregation, to correct LLMs' imbalanced evaluation preference when critiquing math solutions.

Problem

LLM critics for multi-step math reasoning are unreliable; they judge lower-perplexity solutions as correct and show imbalanced evaluation preference, especially when judging solutions from other models.

Approach

Use perplexity as a modulation signal in GRPO advantages: upweight high-perplexity predicted-correct and low-perplexity predicted-wrong samples (counter-preference trajectories), and aggregate loss separately per ground-truth class.

Key results

• On OPS, perplexity-aware GRPO improves average accuracy to 79.79 (Qwen subset-averaged setting) vs 74.76 for vanilla GRPO and reduces average |BI| to 17.14 vs 35.24.
• On ProcessBench, perplexity-aware GRPO reaches average F1 42.24 on Qwen2-7B-Instruct vs 37.95 for vanilla GRPO, and 42.56 vs 38.91 on LLaMA3.1-8B-Instruct.
• Perplexity-aware GRPO achieves the best ProcessBench result in 11 of 15 dataset-model cases according to the authors.

Surprising findings

• Removing perplexity modulation sharply lowers accuracy (75.77) even though it reduces |BI| to 11.64, suggesting balancing bias alone does not improve performance.
• SFT with GPT-4o step annotations lags behind RL methods on ProcessBench error localization.

Limitations

• Evaluated only on 7-8B models and a single math domain (MATH-derived problems).
• Perplexity-based reweighting relies on a hand-chosen linear modulation and base reward values chosen empirically.
• OPS benchmark and training data are derived from a small set of generator models and filtered with a 72B PRM.

Experimental setup

Datasets: MATH train MATH test ProcessBench

Benchmarks: OPS ProcessBench

Metrics: accuracy balance indicator (FPR minus FNR) F1 on error localization

Baselines: SFT Vanilla GRPO DrGRPO

Builds on: GRPO Dr.GRPO VeRL ProcessBench

Closest papers by meaning

• Building Math Agents with Multi-Turn Iterative Preference Learning (ICLR 2025 · similarity 0.90)
• SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning (ICML 2026 · similarity 0.89)
• Conditional Advantage Estimation for Reinforcement Learning in Large Reasoning Models (ICLR 2026 · similarity 0.89)
• Diversity-Aware Policy Optimization for Large Language Model Reasoning (NeurIPS 2025 · similarity 0.89)
• Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.89)
• SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning (NeurIPS 2025 · similarity 0.89)