Research map / Reinforcement learning / RL from feedback

Offline Preference-Based Trajectory Evaluation

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: Reward modeling versus direct optimization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes offline preference-based trajectory evaluation (LR, RPP, IPP measures comparing time-to-return profiles) that replaces success-based metrics, reducing instance-level tie rates from about 75% to about 35% and improving discriminability, reliability and data efficiency across five agentic benchmarks.

Problem

Offline evaluation of agentic systems collapses trajectories to terminal success, discarding partial progress and timing, which produces many ties, lowers effective sample size, and weakens discrimination between systems.

Approach

Compare trajectories directly with temporal-preference measures (Lexicographic Return, Return-Paired Preference, Interval-Paired Preference) over time-to-return profiles, without temporal discounting or extra hyperparameters, and aggregate pairwise preferences with a Bradley-Terry model.

Key results

• Instance-level tie rate falls from 74.9% under success rate to roughly 34-35% for LR, RPP and IPP.
• On SGRL oracle recovery, LR, RPP and IPP reach 94-96% rank accuracy versus 12.8% for SR and 91.8% for SPL; under FDR correction RPP detects 63.2% of true preferences as significant versus 48.1% for SPL.
• Under FDR correction RPP shows significant differences in 78.4% of model pairs versus 58.5% for SR, with zero false positives on same-model pairs.

Surprising findings

• Success rate and partial return achieve only 12.8% oracle rank accuracy because they cannot distinguish successful agents differing only in how they reach the goal.
• Separately reporting success rate and mean time-to-success can produce aggregate order reversals relative to per-task comparisons.

Limitations

• Assumes intermediate returns reflect meaningful progress; noisy or artificially dense subgoal annotations may amplify artifacts.
• Temporal preference may not be a desirable system property in some domains.
• Strongest oracle analyses rely on synthetic gridworld environments, so real-world validity remains to be shown.

Experimental setup

Datasets: AgentBoard OpenHands-Index TheAgentCompany TALES SGRL (sub-goal RL gridworlds)

Benchmarks: AgentBoard OpenHands-Index TheAgentCompany TALES SGRL

Metrics: tie rate discriminative power oracle rank accuracy split-half reliability leave-one-out sign flip rate discriminative bias data efficiency

Baselines: success rate partial return SPL

Builds on: Bradley-Terry model success weighted by path length information retrieval preference-based measures

Closest papers by meaning

• Proper Scoring Rules for Agentic Uncertainty Quantification (NeurIPS 2026 · similarity 0.87)
• Beyond Suspicious Steps: Ontological Trust in Long-Horizon Agents (NeurIPS 2026 · similarity 0.86)
• Consistency as a Testable Property: Statistical Methods to Evaluate AI Agent Reliability (NeurIPS 2026 · similarity 0.86)
• Comparing Comparisons: Informative and Easy Human Feedback with Distinguishability Queries (ICML 2025 · similarity 0.85)
• Reflective Prompted Policy Optimization: Trajectory-Grounded Revision and Salience Bias (NeurIPS 2026 · similarity 0.85)
• AI Alignment with Changing and Influenceable Reward Functions (ICML 2024 · similarity 0.85)