Research map / Reinforcement learning / Offline RL
Value-Rectified Distillation for Flow-based Offline Reinforcement Learning
Ke Jiang, Wen Jiang, Yoshinobu Kawahara, Xiaoyang Tan
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Addresses out-of-distribution mode averaging when distilling flow-matching behavior policies in offline RL on multi-modal datasets, proposing value-rectified distillation.
Problem
Distilling flow-matching behavior policies in offline RL over multi-modal datasets makes the student policy interpolate between behavioral modes, producing invalid out-of-distribution actions.
Approach
Identifies OOD mode averaging as the failure mode of standard distillation of flow-matching policies and proposes value-rectified distillation to counter it; specific mechanism not stated in the abstract.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace FQL distillation with value-rectified distillation (VRD-R) | FQL · D4RL average over all tasks | normalized score: 81.8 -> 92.8 | Improved |
| replace FQL distillation with VRD-Q | FQL · D4RL average over all tasks | normalized score: 81.8 -> 88.8 | Improved |
| replace FQL with VRD-Q | FQL · OGBench (50 tasks) | success rate: 43.5 -> 56.4 | Improved |
| VRD-Q on hopper-medium vs FQL | VRD-Q · D4RL hopper-medium-expert | normalized score: 105.4 -> 96.4 | Hurt |
| VRD-R geometric reshaping vs VRD-Q on locomotion | VRD · D4RL MuJoCo average | normalized score: 80.8 -> 88.5 | Improved |
| VRD-Q statistical filtering vs VRD-R on long-horizon navigation | VRD · OGBench (50 tasks) average | success rate: 50.8 -> 56.4 | Improved |
| quantile tau from 0.5 or 0.9 to 0.7 in VRD-Q (ablation) | VRD-Q · D4RL halfcheetah-medium | normalized score: 60.4 -> 65.7 (tau 0.5 -> 0.7); 58.1 at tau 0.9 | Improved |
| reduce inference latent scale sigma from 1 to 1e-4 in VRD-R (ablation) | VRD-R · D4RL walker2d-medium | normalized score: 53.9 -> 89.7 | Improved |
| smaller sigma below 1e-4 in VRD-R (ablation) | VRD-R · D4RL halfcheetah-medium | normalized score: 61.7 -> 61.2 (1e-4 -> 1e-5) | No clear effect |
| use VRD instead of FQL and CQL on mixed expert-random datasets | VRD-Q and VRD-R · halfcheetah-mix and walker2d-mix | score vs random-sample ratio (50-90%) | Improved |
Experimental setup
Builds on: flow matching policy distillation
Closest papers by meaning
- Offline Behavior Distillation (NeurIPS 2024 · similarity 0.90)
- DeFlow: Decoupling Behavior-Prior Modeling and Value Maximization for Offline Policy Extraction (NeurIPS 2026 · similarity 0.89)
- Flow Matching for Offline Reinforcement Learning with Discrete Actions (NeurIPS 2026 · similarity 0.89)
- Learning Multimodal One-step Flow Policy via Value-weighted Optimal Transport (NeurIPS 2026 · similarity 0.88)
- Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · similarity 0.88)
- Horizon Adaptive Offline Policy Learning via Value Stitching (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
