Research map / Large language models / Reasoning and chain of thought

Think-J: Learning to Think for Generative LLM-as-a-Judge

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: Routing, pruning and distillation for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Think-J improves generative LLM judges by SFT-initializing judgment reasoning traces on 707 curated preference pairs, then optimizing them with critic-guided offline DPO+SFT or rule-based online GRPO with accuracy, format and preference-strength rewards.

Problem

Generative LLM-as-a-Judge models underperform in judgment accuracy and classifier-based reward models lack interpretability; preference datasets lack thinking-trace annotations to train judges to reason well.

Approach

Initialize judge thinking with a small curated set of R1-annotated traces (with trace clipping), then optimize the reasoning via either a trained critic that builds positive/negative traces for offline DPO+SFT, or GRPO with rule-based rewards that include a preference-strength term.

Key results

• Think-J-Qwen-2.5-32B reaches 90.5 overall on RewardBench, the best among listed judges (GPT-4o 86.7, Claude-3.5-Sonnet 84.2), and 79.8 on RMBench.
• On Qwen2.5-7B-Instruct trained on Helpsteer2-Pref, Think-J (GRPO) reaches 85.3 RewardBench overall vs 77.3 for BT classifier and 75.6 for SynRM.
• Among RL variants on Llama-3-8B, GRPO (84.1) outperforms Reinforce++ (84.0) and PPO (73.2); critic-guided offline DPO reaches 83.2.

Surprising findings

• Adding a margin-based absolute scoring reward caused reward hacking, with scores driven to extreme values.
• Trace clipping of R1 traces improved both accuracy and inference speed; longer thinking did not correlate with better judge accuracy.
• On noisy HH-RLHF groups, a model with high judgment accuracy produced thinking traces with over 50% identifiable errors, suggesting correct verdicts can arise from flawed reasoning.

Limitations

• Preference strength reward requires 3-level strength annotations, which may be unavailable or require a BT classifier to derive.
• Margin-based absolute scoring reward leads to reward hacking and was not solved.
• Thinking traces can be erroneous even when final judgments are correct, so interpretability is not fully reliable.

Experimental setup

Datasets: Skywork-Reward-Preference-80K-v0.2 HelpSteer2-Pref HH-RLHF LIMJ707

Benchmarks: RewardBench RMBench Auto-J-test

Metrics: accuracy agreement rate

Baselines: JudgeLM-33B Prometheus-7b-v2.0 Prometheus-8x7b-v2.0 Llama-3-OffsetBias-8B CompassJudger-32B STE-Llama3.1-70B SynRM CLoud BT Classifier Claude-3.5-Sonnet GPT-4o Gemini-1.5-Pro

Builds on: DeepSeek-R1 GRPO DPO Skywork-Reward-Preference RewardBench

Closest papers by meaning

• J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning (ICLR 2026 · similarity 0.93)
• Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge (ICML 2025 · similarity 0.90)
• Reverse Engineering Human Preferences with Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
• Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models (NeurIPS 2025 · similarity 0.88)
• GRAM: A Generative Foundation Reward Model for Reward Generalization (ICML 2025 · similarity 0.87)
• Critique-Coder: Enhancing Coder Models by Critique Reinforcement Learning (ICLR 2026 · similarity 0.87)