Research map / Vision-language and multimodal / Robot action and navigation
What Matters to You? Towards Visual Representation Alignment for Robot Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RAPL, which fine-tunes pre-trained visual encoders using only human video-triplet preference rankings and uses optimal-transport feature matching as the robot reward, showing sample-efficient and cross-embodiment preference-aligned behavior in simulation.
Problem
Robots relying on pre-trained visual representations often learn rewards that capture spurious correlations and do not match end-user preferences; existing fine-tuning uses proxy tasks (dynamics prediction, cycle-consistency, behavior cloning) that bypass human input.
Approach
Align the robot's visual embedding with the human's via preference-based triplet rankings of videos (Bradley-Terry likelihood over distances), with no action labels, and then use an optimal-transport (Sinkhorn) feature-matching reward computed in the aligned space; only the last linear layer is fine-tuned.
Key results
- On X-Magical avoid, RAPL reaches 70% success in 10 epochs versus 100 for the ground-truth reward oracle, and matches GT at about 80% success.
- In Franka manipulation with 150 preference queries, RAPL reaches about 70% success while baselines stay near or below 10%; RAPL with 25 queries reaches about 45%.
- Spearman correlation with ground-truth reward on Franka Group is 0.59 for RAPL versus 0.38 for RLHF and near zero or negative for other visual baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a preference-aligned representation with optimal-transport reward instead of direct preference reward prediction | Robot manipulation policy learning · Franka Group | success rate: 75% | Helped |
| Increase RLHF preference queries from 150 to 300 (ablation) | RLHF · Franka Group | policy performance | Helped |
| Reduce RAPL preference queries to 25 (ablation) | RAPL · Franka Group | success rate: 45% | Mixed |
| Transfer the representation learned on the short-stick agent to the medium-stick agent | RAPL · X-MAGICAL grouping, cross-embodiment | success rate: improves by 33%; succ. rate ≈ 80% | Helped |
| Use RAPL rather than RLHF | Visual reward learning · Franka Group | Spearman's correlation: 0.38 -> 0.59 | Helped |
| Use RAPL rather than RLHF | Visual reward learning · Kuka Group | Spearman's correlation: 0.31 -> 0.47 | Helped |
| Use RAPL rather than RLHF | Visual reward learning · Franka Clutter | Spearman's correlation: 0.26 -> 0.61 | Helped |
| Use RAPL rather than TCC visual reward | X-MAGICAL grouping with progress-only preference · X-MAGICAL grouping | success rate | No clear effect |
Surprising findings
- RLHF, despite a similar objective, fails in the more realistic manipulation environment, with attention focused on the robot arm and background (spurious correlations).
- Transferring the short-stick representation zero-shot to a medium-stick embodiment improved RAPL's grouping success by about 33%.
Limitations
- Evaluated only with a simulated human using ground-truth reward; no real user feedback or hardware experiments.
- Offline fine-tuning only; no online preference-based representation updates.
- Only the final linear layer is fine-tuned, with all encoders frozen during policy learning.
Experimental setup
Datasets: X-Magical IsaacGym Franka grouping/clutter simulated videos IsaacGym Kuka videos Ego4D (for baseline pre-training) ImageNet (for baseline)
Benchmarks: X-Magical avoid X-Magical grouping IsaacGym Franka grouping IsaacGym Franka clutter cross-embodiment X-Magical cross-embodiment Kuka manipulation
Metrics: task success rate Spearman correlation with ground-truth reward sample efficiency (epochs to threshold success)
Baselines: GT (ground-truth reward oracle) RLHF (preference reward learning) TCC MVP-OT Fine-tuned MVP-OT R3M-OT ImageNet-OT TCC-OT
Builds on: TCC (temporal cycle-consistency) Sinkhorn algorithm Bradley-Terry model MVP (masked visual pre-training) soft actor-critic
Closest papers by meaning
- SYNAPSE: SYmbolic Neural-Aided Preference Synthesis Engine (AAAI 2025 · similarity 0.89)
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.89)
- RoboAlign-R1: Distilled Multimodal Reward Alignment for Robot Video World Models (NeurIPS 2026 · similarity 0.87)
- PEARL: Zero-shot Cross-task Preference Alignment and Robust Reward Learning for Robotic Manipulation (ICML 2024 · similarity 0.87)
- Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformers (NeurIPS 2024 · similarity 0.87)
- DecisionNCE: Embodied Multimodal Representations via Implicit Preference Learning (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.