Research map / Reinforcement learning / Imitation learning
Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Successor Feature Matching (SFM), a non-adversarial state-only IRL method that matches expert successor features via policy gradient with learned base features, evaluated on DMControl from a single demonstration.
Problem
Inverse RL is usually a costly, unstable min-max game between reward learning and RL, and most state-only methods still rely on adversarial training; action labels are also often unavailable in demonstrations.
Approach
Exploit the linear factorization of returns via successor features so the witness reward is the (normalized) difference between expert and agent successor features; the policy is then optimized directly by policy gradient on this gap, with no explicit reward model or adversary, and base features learned jointly with the policy, enabling state-only imitation.
Key results
- SFM outperforms competitors by 16% on mean normalized returns across 10 DMControl tasks with a single demonstration, without expert action labels.
- SFM with TD3 performs nearly as well as with TD7, whereas state-only baselines degrade considerably under TD3.
- FDM base features performed best among random, IDM, Hilbert, AE, adversarial and FDM features on DMC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace adversarial or action-label-dependent imitation baselines with state-only SFM | single-demonstration imitation on DMControl · DMControl suite, 10 tasks | mean normalized return: 16% | Helped |
| use TD3 instead of TD7 as policy optimizer (ablation) | SFM · DMControl suite | return | No clear effect |
| use TD3 instead of TD7 as policy optimizer (ablation) | state-only adversarial baselines · DMControl suite | return | Hurt |
| use forward dynamics model features rather than other tested feature functions (ablation) | SFM · DMControl suite | return | Helped |
| use inverse dynamics model features instead of FDM features (ablation) | SFM · quadruped tasks | return | Hurt |
| use random features instead of FDM features (ablation) | SFM · DMControl suite | return | Mixed |
| use autoencoder or adversarial features instead of FDM features (ablation) | SFM · DMControl suite | return | Hurt |
| use an entropy-regularized stochastic policy instead of deterministic policy optimizers (ablation) | SFM · DMControl suite | return | Mixed |
Surprising findings
- SFM surpasses agents that have access to expert action labels (BC and IQ-Learn) despite learning from states only.
- Adversarially trained base features were not better than the simpler FDM features.
Limitations
- Does not alleviate the exploration problem inherent in IRL methods.
- Evaluated only on DMControl tasks with a single demonstration; the approach relies on learned base features whose richness is not guaranteed.
Experimental setup
Datasets: DeepMind Control suite demonstrations (single expert trajectory per task)
Benchmarks: DeepMind Control Suite (10 tasks: Cheetah, Quadruped, Walker)
Metrics: mean normalized return returns per task RLiable interquartile mean and optimality gap with 95% boots
Baselines: Behavior Cloning IQ-Learn OPOLO state-only MM GAIfO
Builds on: successor features TD3 TD7 moment matching IQ-Learn
Closest papers by meaning
- Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective (ICML 2024 · similarity 0.91)
- Recent Advancements in Inverse Reinforcement Learning (AAAI 2024 · similarity 0.91)
- Trust Region Reward Optimization and Proximal Inverse Reward Optimization Algorithm (NeurIPS 2025 · similarity 0.91)
- Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithms (ICML 2024 · similarity 0.90)
- Hybrid Inverse Reinforcement Learning (ICML 2024 · similarity 0.90)
- Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.