Research map / Large language models / Reasoning and chain of thought

J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes J1, an online-RL (GRPO) recipe that trains thinking LLM judges from 22K synthetic preference pairs by converting pairwise judging into a verifiable task, with consistency rewards and a pointwise/multitask variant, reaching SOTA on several reward-model benchmarks at 8B/32B/70B.

Problem

LLM-as-a-Judge models need reasoning before verdicts, but judgment quality for both verifiable and non-verifiable prompts is hard to optimize directly, and pairwise judges suffer from position bias.

Approach

Convert all judgment tasks (verifiable and non-verifiable) into verifiable tasks using synthetic preference pairs, reward verdict correctness and position-consistency under both response orders with online GRPO, and train a pointwise judge via distant supervision from pairwise labels, then unify both in one multitask model.

Key results

• J1-Qwen-32B-MultiTask reaches 76.8 overall on PPE Correctness vs 66.5 for base Qwen3-32B (thinking) and 70.2 for EvalPlanner-Llama-70B.
• J1-Qwen-32B-MultiTask scores 93.6 on RewardBench, the top score among listed models including DeepSeek-R1 (90.6) and o3 (86.4).
• J1-Llama-8B improves RM-Bench to 73.4 from 54.0 for base Llama-3.1-8B-Instruct, trained on the same 22K synthetic pairs.

Surprising findings

• Pointwise judges trained only with pairwise supervision achieve higher position-consistent accuracy than the corresponding pairwise judges (e.g., 65.0 vs 61.2 for 8B).
• Predicting only the verdict performs as well as variants that also output scores in the pairwise setting.

Limitations

• Relies on synthetic preference pairs whose rejected responses come from LLM-generated noisy instructions, which may not match real distributions.
• Pointwise judges are more prone to ties, and position-consistent accuracy remains modest in absolute terms (about 61-67 for pairwise).

Experimental setup

Datasets: WildChat MATH PPE RewardBench JudgeBench RM-Bench FollowBenchEval

Benchmarks: PPE Correctness PPE Preference RewardBench RM-Bench JudgeBench FollowBenchEval

Metrics: accuracy position-consistent accuracy verdict flip rate tie rate

Baselines: EvalPlanner DeepSeek-GRM-27B DeepSeek-R1-Distilled-Llama DeepSeek-R1 OpenAI-o1-mini OpenAI-o3 Skywork-Reward-Gemma-2-27B DeepSeek-BTRM-27B

Builds on: GRPO DeepSeek-R1 DPO EvalPlanner verl

Closest papers by meaning

• Think-J: Learning to Think for Generative LLM-as-a-Judge (AAAI 2026 · similarity 0.93)
• Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge (ICML 2025 · similarity 0.91)
• Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning (ICLR 2026 · similarity 0.90)
• Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.89)
• JudgeBench: A Benchmark for Evaluating LLM-Based Judges (ICLR 2025 · similarity 0.89)
• Thinkless: LLM Learns When to Think (NeurIPS 2025 · similarity 0.89)