Research map / Reinforcement learning / RL from feedback
STAIR: Addressing Stage Misalignment through Temporal-Aligned Preference Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STAIR, a PbRL method that learns contrastive temporal distance as an on-policy stage approximator and selects queries whose segments lie in the same stage, improving feedback efficiency on multi-stage robotic tasks.
Problem
Preference-based RL in multi-stage tasks suffers from stage misalignment: comparing segments from different stages yields uninformative human feedback and inefficient reward/policy learning.
Approach
Learn a successor-distance (quasimetric) temporal distance via symmetrized infoNCE contrastive learning, convert it to a quadrilateral segment-level stage distance, and score queries by combining stage alignment with ensemble-disagreement uncertainty to select stage-aligned comparisons.
Key results
- STAIR reaches success rates close to 100% in most multi-stage MetaWorld and RoboDesk tasks, outperforming PEBBLE, RUNE, MRN, RIME and QPA with faster convergence.
- On single-stage DMControl tasks STAIR outperforms PEBBLE and is competitive with other baselines.
- Human labelers judged a higher proportion of STAIR-selected queries as stage-aligned than PEBBLE-selected queries.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use STAIR's temporal-distance stage-aligned query selection instead of baseline query selection | PbRL on MetaWorld and RoboDesk multi-stage tasks · MetaWorld and RoboDesk | success rate | Helped |
| Use STAIR instead of PEBBLE | PbRL on DMControl single-stage tasks · DMControl | episodic return | Helped |
| Use STAIR-generated queries instead of PEBBLE-generated queries | Human assessment of query stage alignment · Human labeling experiment | ratio of queries judged stage-aligned | Helped |
| Use timestep as the stage approximation and Interval Span Ratio for segment distance (ablation) | STAIR · Ablation study on benchmark tasks | task performance | Hurt |
| Use temporal distance from the current state to the trajectory's initial state as the stage approximation, with Interval Span Ratio (ablation) | STAIR · Ablation study on benchmark tasks | task performance | Hurt |
| Reduce temporal-distance update frequency (increase K_SD) (ablation) | STAIR · Ablation study on benchmark tasks | task performance | Hurt |
| Vary c_stage and c_state across the tested configurations (ablation) | STAIR · Benchmark tasks | task performance | No clear effect |
| Use STAIR instead of PEBBLE with different numbers of query feedback | PbRL on MetaWorld door-open · MetaWorld door-open | task performance | Helped |
| Select comparisons within the same stage rather than across stages | PbRL in the abstract MDP · Abstract MDP with stage reward bias | learning efficiency | Helped |
Surprising findings
- Stage-aligned selection helps even on single-stage DMControl tasks, which the authors attribute to implicit curriculum learning.
Limitations
- Quadrilateral distance only assesses pairwise segment differences, limiting applicability to other preference formats.
- Stage-approximation quality depends on update frequency of the temporal distance model; less frequent updates degrade performance.
Experimental setup
Datasets: MetaWorld DMControl RoboDesk
Benchmarks: MetaWorld DMControl RoboDesk
Metrics: success rate episodic return ratio of stage-aligned queries (human study)
Baselines: PEBBLE RUNE MRN RIME QPA SAC
Builds on: PEBBLE successor distance SAC BPref
Closest papers by meaning
- Query-Policy Misalignment in Preference-Based Reinforcement Learning (ICLR 2024 · similarity 0.89)
- CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries (ICML 2025 · similarity 0.88)
- Comparing Comparisons: Informative and Easy Human Feedback with Distinguishability Queries (ICML 2025 · similarity 0.88)
- AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model (ICLR 2024 · similarity 0.87)
- Hindsight PRIORs for Reward Learning from Human Preferences (ICLR 2024 · similarity 0.87)
- Robust Reward Alignment via Hypothesis Space Batch Cutting (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.