Research map / Large language models / Reasoning and chain of thought

UniRRM: Unified Reasoning Reward Models Across Languages and Evaluation Paradigms

ICML 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MixReward (64.5K multilingual pairwise/listwise preference data across 103 languages) and UniRRM, a Qwen3-based generative reasoning reward model trained with SFT then GRPO that unifies pair-wise, list-wise and point-wise judging with adaptive rubrics.

Problem

Generative reward models for open-ended tasks are limited by static rubrics, fragmented pairwise/pointwise/listwise evaluation paradigms, and English-centric training data.

Approach

A staged reasoning chain (analysis, then adaptive rubric generation split into task-generic and instruction-specific criteria, then evaluation) produces a single output that yields pairwise, listwise and pointwise judgments, trained by distillation from GPT-OSS-120B followed by GRPO with format, rubric-quality and accuracy rewards.

Key results

• UniRRM-14B reaches 0.868 average pairwise accuracy, slightly above its GPT-OSS-120B teacher (0.865), and scores 0.885 on MM-Eval.
• UniRRM-14B achieves 0.771 point-wise average accuracy, above M-Prometheus-14B (0.723), without point-wise training.
• In RL on GSM8K and Math-500 with Qwen3-0.6B, UniRRM-8B as reward model gives 77.4 on Math-500 vs 74.2 with Skywork-Reward-Llama-3.1-8B.

Surprising findings

• UniRRM-14B slightly exceeds its GPT-OSS-120B teacher on average pairwise accuracy despite being distilled from it.
• Reasoning in the target language instead of English causes only a marginal drop (0.863 to 0.851).

Limitations

• Relies on GPT-OSS-120B teacher distillation and teacher-based rubric critic, so quality is bounded by the teacher.
• Translated data is not strictly linguistically verified; only preference consistency is checked.
• Performance on JudgeBench and some multilingual Chat Hard subsets remains well below the best judges.

Experimental setup

Datasets: MixReward UltraFeedback WildChat Arena-Human-Preference-140K Tulu-3-Pref-Personas-Instruction-Following MATH12k HelpSteer3 MATH-500-multilingual HumanEval-XL PKU-SafeRLHF GSM8K Math-500

Benchmarks: RewardBench M-RewardBench MM-Eval JudgeBench RewardBench v2 GSM8K Math-500

Metrics: accuracy

Baselines: GPT-4o Qwen3-8B Qwen3-14B GPT-OSS-120B Skywork-Reward-V2-Llama-3.1-8B Skywork-Reward-Gemma-2-27B-v0.2 RubricRM RM-R1-Distilled-Qwen-32B JudgeLRM-7B M-Prometheus-14B mR3 RewardAnything

Builds on: GRPO RubricRM RM-R1 RewardAnything LlamaFactory VeRL

Closest papers by meaning

• RM-R1: Reward Modeling as Reasoning (ICLR 2026 · similarity 0.92)
• Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models (NeurIPS 2025 · similarity 0.90)
• RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style (ICLR 2025 · similarity 0.89)
• Reward Reasoning Models (NeurIPS 2025 · similarity 0.89)
• A Simple "Motivation" Can Enhance Reinforcement Finetuning of Large Reasoning Models (ICLR 2026 · similarity 0.89)
• VerifyBench: Benchmarking Reference-based Reward Systems for Large Language Models (ICLR 2026 · similarity 0.89)