Research map / Reinforcement learning / Imitation learning
SEABO: A Simple Search-Based Method for Offline Imitation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SEABO, an unsupervised reward-labeling method that scores each unlabeled transition by its distance to the nearest neighbor in a single expert demonstration (KD-tree), then runs any offline RL algorithm (IQL, TD3_BC) on the relabeled data.
Problem
Offline imitation learning needs reward labels on unlabeled offline data, but hand-crafted rewards are costly and existing reward-learning or distribution-matching IL methods are unstable, complex, or need discriminators.
Approach
Assign reward by a squashing function of the Euclidean distance from each unlabeled (s,a,s') transition to its nearest neighbor in the expert demonstration, found via a KD-tree; no reward model, discriminator, or optimal transport is trained.
Key results
- IQL+SEABO beats IQL with ground-truth rewards on 6 of 9 MuJoCo datasets and reaches a total score of 713.3 vs 692.4 for IQL with true rewards.
- With a single expert trajectory, IQL+SEABO outperforms IQL+OTR on AntMaze (total 400.8 vs 384.0) and Adroit (153.6 vs 122.2 total score).
- SEABO in the state-only regime achieves a total score of 707.6, above LobsDICE (531.8) and OTR (685.6).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SEABO reward labels instead of ground-truth rewards | IQL · D4RL MuJoCo locomotion (9 tasks) | total normalized score: 692.4 -> 713.3 | Helped |
| Use SEABO reward labels instead of ground-truth rewards | TD3_BC · D4RL MuJoCo locomotion (9 tasks) | total normalized score: 692.3 -> 723.5 | Helped |
| Use SEABO reward labels instead of IQL+OTR | IQL · AntMaze-v0 (6 tasks) | total normalized score: 384.0 -> 400.8 | Helped |
| Use SEABO reward labels instead of IQL+OTR | IQL · Adroit-v0 (8 tasks) | total normalized score: 122.2 -> 153.6 | Helped |
| Replace KD-tree with Ball-tree search (ablation) | SEABO · D4RL MuJoCo locomotion (9 tasks) | total normalized score: 713.3 -> 712.1 | No clear effect |
| Replace KD-tree with HNSW search (ablation) | SEABO · D4RL MuJoCo locomotion (9 tasks) | total normalized score: 713.3 -> 386.4 | Hurt |
| Increase expert demonstrations from K=1 to K=10 (ablation) | IQL+SEABO · D4RL MuJoCo locomotion (9 tasks) | total normalized score: 713.3 -> 716.1 | Helped |
| Use expert observations only instead of state-action demonstrations (ablation) | SEABO with IQL · D4RL MuJoCo locomotion (9 tasks) | total normalized score: 713.3 -> 707.6 | Mixed |
| Increase expert demonstrations from K=1 to K=10 (ablation) | IQL+SEABO · AntMaze-v0 (6 tasks) | total normalized score: 400.8 -> 399.2 | Mixed |
| Increase expert demonstrations from K=1 to K=10 (ablation) | IQL+SEABO · Adroit-v0 (8 tasks) | total normalized score: 153.6 -> 145.2 | Hurt |
| Set reward scale α across {1,5,10} (ablation) | IQL+SEABO · D4RL locomotion datasets | normalized score | No clear effect |
| Vary distance coefficient β across {0.1,0.5,1,5} (ablation) | IQL+SEABO · D4RL locomotion datasets | normalized score | Mixed |
Surprising findings
- SEABO with a naive Euclidean KD-tree matches or exceeds IQL trained on ground-truth rewards in aggregate, despite using only one expert trajectory and no learned reward model.
- HNSW as the nearest-neighbor search backend performs poorly on many tasks (total 386.4), so the choice of search algorithm matters.
Limitations
- Performance is sensitive to the reward hyperparameters alpha and beta, which are task-dependent.
- SEABO does not perform strongly on high-precision tasks such as IKEA furniture assembly.
- Relies on Euclidean distance in raw state-action space, so feature scaling across dimensions can affect the reward.
Experimental setup
Datasets: D4RL MuJoCo locomotion (halfcheetah, hopper, walker2d) D4RL AntMaze D4RL Adroit D4RL Kitchen
Benchmarks: D4RL MuJoCo locomotion D4RL AntMaze D4RL Adroit D4RL Kitchen
Metrics: normalized score total score
Baselines: IQL (ground-truth reward) TD3_BC BC 10%BC ORIL UDS OTR SQIL DemoDICE SMODICE PWIL LobsDICE
Builds on: IQL TD3_BC PWIL OTR
Closest papers by meaning
- Enhancing Online Reinforcement Learning with Meta-Learned Objective from Offline Data (AAAI 2025 · similarity 0.90)
- How to Leverage Diverse Demonstrations in Offline Imitation Learning (ICML 2024 · similarity 0.90)
- OLLIE: Imitation Learning from Offline Pretraining to Online Finetuning (ICML 2024 · similarity 0.90)
- Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning (NeurIPS 2024 · similarity 0.90)
- Imitation Learning from Observation with Automatic Discount Scheduling (ICLR 2024 · similarity 0.89)
- Reward-free World Models for Online Imitation Learning (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.