Research map / Reinforcement learning / Policy gradient and actor-critic
SF-DQN: Provable Knowledge Transfer using Successor Feature for Deep Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Provides the first convergence and generalization analysis for successor feature learning with deep Q-networks (SF-DQN) and GPI in transfer RL, with theory supporting faster convergence and better transfer than DQN.
Problem
Theoretical guarantees are lacking for successor feature (SF) learning with deep neural network function approximation and GPI-based transfer in multi-task RL sharing transition dynamics but differing rewards.
Approach
Decompose Q-functions into successor features (parameterized by a one-hidden-layer-style DNN) times task-specific reward weights, learn them alternately with mini-batch SGD over a replay buffer, and transfer via GPI across prior tasks; derive convergence rates of 1/T with generalization bounds and show improvement over DQN transfer by a factor related to gamma.
Key results
- With GPI, SF-DQN outperforms SF-DQN without GPI when tasks are similar: normalized reward 0.986±0.007 vs 0.942±0.004 at ||w1*-w2*||=0.01.
- Theory gives a 1/T convergence rate for SF-DQN with generalization error scaling with ||w1(0)-w1*||, and transfer error for SF-DQN is better than DQN transfer by a factor of (1+gamma)/2.
- Experiments show SF-DQN achieves higher average reward on task 2 than DQN when transfer begins.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add GPI (ablation) | SF-DQN · Synthetic RL, task relevance ||w1* - w2*|| = 0.01 | normalized average reward: 0.942 ± 0.004 -> 0.986 ± 0.007 | Helped |
| add GPI (ablation) | SF-DQN · Synthetic RL, task relevance ||w1* - w2*|| = 0.1 | normalized average reward: 0.911 ± 0.013 -> 0.965 ± 0.007 | Helped |
| add GPI (ablation) | SF-DQN · Synthetic RL, task relevance ||w1* - w2*|| = 1 | normalized average reward: 0.813 ± 0.009 -> 0.827 ± 0.008 | Helped |
| add GPI (ablation) | SF-DQN · Synthetic RL, task relevance ||w1* - w2*|| = 10 | normalized average reward: 0.707 ± 0.011 -> 0.717 ± 0.012 | No clear effect |
Surprising findings
- GPI gives no benefit when tasks are dissimilar (large weight distance), with both variants performing similarly.
Limitations
- Theory requires good initialization near ground truth and one-hidden-layer style analysis extended to multi-layer networks with strong assumptions (realizability, bounded policy distribution shift).
- Experiments are limited to a synthetic environment in the main text; real benchmark details are deferred to appendix.
Experimental setup
Datasets: synthetic RL environment real-world RL benchmarks (unspecified in main text)
Benchmarks: synthetic RL environment
Metrics: normalized average reward accumulated reward
Baselines: DQN Q-learning SF-DQN without GPI
Builds on: successor features generalized policy improvement deep Q-network successor representation
Closest papers by meaning
- Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement (ICML 2018 · similarity 0.91)
- A Transfer Approach Using Graph Neural Networks in Deep Reinforcement Learning (AAAI 2024 · similarity 0.86)
- Model-Based Transfer Learning for Contextual Reinforcement Learning (NeurIPS 2024 · similarity 0.85)
- Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective (ICML 2025 · similarity 0.85)
- Learning Successor Features the Simple Way (NeurIPS 2024 · similarity 0.85)
- Transfer Q-Learning with Composite MDP Structures (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.