Research map / Large language models / Reasoning and chain of thought

Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domains

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Curates a 2.5M-sample multi-task, reasoning-focused evaluator dataset and trains FARE-8B and FARE-20B generative evaluators via iterative rejection-sampling SFT, reporting strong results on pairwise, step-level, verification, reranking, and RL-verifier benchmarks.

Problem

Generative automatic evaluators are usually trained on small, single-task data or with costly RL, while evaluators are needed across multiple evaluation tasks and reasoning domains.

Approach

Scale multi-task evaluator training data to 2.5M samples (existing plus synthetic generate-then-grade and programmatic error injection) and train with a semi-online iterative rejection-sampling SFT loop starting from an existing post-trained LLM (gpt-oss-20B or a cold-started Qwen3-8B), with direct-judgment data and a per-batch difficulty curriculum.</non-parameter>

Key results

• FARE-20B reaches 84.4 overall F1 on ProcessBench, nearly matching GPT-5 (84.6) and beating Qwen2.5-Math-72B-PRM (78.3).
• Using FARE-20B as GRPO verifier raises downstream average from 34.3 to 45.2; FARE-8B and FARE-20B give 8.8% and 14.1% relative gains over string-matching verifiers.
• FARE-20B achieves near-oracle best-of-10 reranking on MATH with Llama-3.1-8B generator (50.83 vs oracle 53.47).

Surprising findings

• Direct judgment without critique or CoT degrades FARE-20B but improves FARE-8B.
• Scaling training data steadily improves positional consistency of pairwise judges without explicit debiasing.

Limitations

• Qwen3-8B cold-start needed because post-trained Qwen3 was considered hard to finetune.
• FARE-20B direct-judgment without CoT degrades performance, suggesting strong CoT prior in gpt-oss.
• Some baselines' numbers taken from original papers rather than re-run under identical settings.

Experimental setup

Datasets: ActPRM Beavertails Preference Code Preference Pairs DeepScaleR Folio FoVer HelpSteer HelpSteer2 HelpSteer3 HH-RLHF Harmless LAMP MATH

Benchmarks: JudgeBench ReasoningJudgeBench PPE Correctness RM-Bench When2Call ProcessBench VerifyBench VerifyBench-Hard BiGGen Bench FLASK JETTS (MATH, CHAMP, MBPP+, BigCodeBench) MMLU-Pro

Metrics: consistent accuracy F1 accuracy Pearson correlation best-of-10 reranking accuracy Avg@32

Baselines: RISE-Judge-7B EvalPlanner-8B J1-8B RM-R1-14B CompassJudger-7B Atla Selene 8B J1-70B GPT-5 StepWiser-7B Qwen2.5-Math-72B-PRM DeepSeek-GRM-27B SFR-Judge-70B

Builds on: STaR RAFT Self-Taught Evaluator EvalPlanner gpt-oss-20B Qwen3-8B-Base

Closest papers by meaning

• Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators (ICML 2025 · similarity 0.88)
• RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic Sampling (ICLR 2026 · similarity 0.87)
• Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learning (ICLR 2026 · similarity 0.86)
• Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge (ICML 2025 · similarity 0.86)
• Critique-Coder: Enhancing Coder Models by Critique Reinforcement Learning (ICLR 2026 · similarity 0.86)
• Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models (NeurIPS 2025 · similarity 0.86)