Research map / Reinforcement learning / Imitation learning
RLfOLD: Reinforcement Learning from Online Demonstrations in Urban Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RLfOLD, an RL-from-demonstrations method using an online rule-based expert in CARLA, a dual-standard-deviation SAC policy, and uncertainty-triggered expert guidance, reaching state-of-the-art NoCrash success with a 0.65M-parameter single-camera encoder.
Problem
Offline expert demonstrations used in RLfD create a distribution gap with the training environment, limiting performance in urban autonomous driving.
Approach
Collect demonstrations online from a privileged rule-based expert into a single replay buffer, use a policy that outputs two standard deviations (one for RL exploration, one for IL log-likelihood), and let the expert take over actions when the policy's exploration uncertainty exceeds a threshold.
Key results
- RLfOLD achieves an average NoCrash success rate of 89%, versus 87% for WOR and 83% for GRIAD, the best prior methods.
- Using a 0.65M-parameter single-camera encoder, about 3% of the size of state-of-the-art encoders.
- Ablation: removing the two-SD design drops success from 86% to 64%; removing uncertainty-guided expert guidance drops it to 72%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add online demonstrations and integrate IL with RL (ablation) | RLfOLD · NoCrash, regular task, testing conditions | success rate (%): 52±4 -> 86±4 | Helped |
| use two standard deviations, with separate deviations for IL and RL (ablation) | RLfOLD · NoCrash, regular task, testing conditions | success rate (%): 64±10 -> 86±4 | Helped |
| add uncertainty-based online expert guidance during exploration (ablation) | RLfOLD · NoCrash, regular task, testing conditions | success rate (%): 72±2 -> 86±4 | Helped |
| replace uncertainty-based guidance with expert guidance at a fixed probability of 0.3 (ablation) | RLfOLD · NoCrash, regular task, testing conditions | success rate (%): 80±3 -> 86±4 | Hurt |
| use RLfOLD instead of the strongest reported baseline | NoCrash benchmark · NoCrash, empty, Town 1, training weather | success rate (%): 98 -> 100 | Helped |
| use RLfOLD instead of the strongest reported baseline | NoCrash benchmark · NoCrash, regular, Town 1, training weather | success rate (%): 100 -> 94 | Hurt |
| use RLfOLD instead of the strongest reported baseline | NoCrash benchmark · NoCrash, dense, Town 1, training weather | success rate (%): 96 -> 90 | Hurt |
| use RLfOLD instead of the strongest reported baseline | NoCrash benchmark · NoCrash, empty, Town 2, training weather | success rate (%): 94 -> 100 | Helped |
| use RLfOLD instead of the strongest reported baseline | NoCrash benchmark · NoCrash, regular, Town 2, training weather | success rate (%): 93 -> 92 | Hurt |
| use RLfOLD instead of the strongest reported baseline | NoCrash benchmark · NoCrash, dense, Town 2, training weather | success rate (%): 78 -> 80 | Helped |
| use RLfOLD instead of the strongest reported baseline | NoCrash benchmark · NoCrash, empty, Town 1, testing weather | success rate (%): 60 -> 96 | Helped |
| use RLfOLD instead of the strongest reported baseline | NoCrash benchmark · NoCrash, regular, Town 1, testing weather | success rate (%): 90 -> 84 | Hurt |
Surprising findings
- Using the RL policy's steering rather than the expert's steering gave better results, so the rule-based expert only supplies target speed.
Limitations
- Online expert is a simple rule-based heuristic set, and overall performance depends on expert quality.
- Only evaluated on CARLA NoCrash; generality to other applications untested.
Experimental setup
Datasets: CARLA simulator online demonstrations (Town 1 training)
Benchmarks: CARLA NoCrash
Metrics: success rate route completion collisions per km agent blocked
Baselines: IAs CADRE CILRS LBC GRIAD WOR RL baseline
Builds on: Soft Actor-Critic RLAD GRIAD DAGGER CADRE
Closest papers by meaning
- Learning from Ambiguous Demonstrations with Self-Explanation Guided Reinforcement Learning (AAAI 2024 · similarity 0.87)
- RLFTSim: Realistic and Controllable Multi-Agent Traffic Simulation via Reinforcement Learning Fine-Tuning (CVPR 2026 · similarity 0.87)
- Dual RL: Unification and New Methods for Reinforcement and Imitation Learning (ICLR 2024 · similarity 0.87)
- RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS-based Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- Enhancing Online Reinforcement Learning with Meta-Learned Objective from Offline Data (AAAI 2025 · similarity 0.87)
- Video-Enhanced Offline Reinforcement Learning: A Model-Based Approach (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.