Research map / Label-efficient and robust learning / Semi-supervised learning
Uncertainty-Guided Never-Ending Learning to Drive
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ∞-Driver, a semi-supervised never-ending self-training framework that pseudo-labels unlabeled driving video with an uncertainty-aware ensemble of inverse dynamics models and trains a conditional imitation learning driving policy with uncertainty-guided replay.
Problem
Continually adapting vision-based end-to-end driving policies from unlabeled, non-stationary streaming video (including web-scale YouTube data) without ground-truth labels, while avoiding forgetting and noise from pseudo-labels.
Approach
Formulates incremental policy learning as distillation from an ensemble inverse dynamics teacher whose epistemic uncertainty (ensemble std and matrix Fisher entropy) drives temporal-consistency relabeling, noisy-sample filtering, and an uncertainty-seeking episodic replay buffer.
Key results
- On city-sequence incremental learning with buffer 3,000, ∞-Driver reaches final average loss 0.66 vs. 0.76 (DER), 1.29 (Rainbow), 0.85 (PuriDivER).
- On open web training over 10M YouTube images, ∞-Driver achieves ADE 1.13 m vs. 1.18 (DER and PuriDivER) and 1.24 (Rainbow).
- Consistently lowest forgetting across buffer sizes 3,000/6,000/9,000 in Table 1 (e.g., -0.03 at buffer 9,000 vs. -0.08 for PuriDivER).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ∞-Driver instead of DER | Cross-city incremental learning, buffer 3,000 · Cross-city incremental learning, buffer 3,000 | Average Loss (L−1): 0.76 -> 0.66 | Helped |
| Use ∞-Driver instead of DER | Cross-city incremental learning, buffer 3,000 · Cross-city incremental learning, buffer 3,000 | Forgetting (F−1): -0.14 -> -0.07 | Helped |
| Use ∞-Driver instead of DER | Cross-city incremental learning, buffer 3,000 · Cross-city incremental learning, buffer 3,000 | Intransigence (I−1): -0.38 -> -0.35 | Hurt |
| Use ∞-Driver instead of PuriDivER | Cross-city incremental learning, buffer 6,000 · Cross-city incremental learning, buffer 6,000 | Average Loss (L−1): 0.84 -> 0.61 | Helped |
| Use ∞-Driver instead of PuriDivER | Cross-city incremental learning, buffer 6,000 · Cross-city incremental learning, buffer 6,000 | Forgetting (F−1): -0.11 -> -0.04 | Helped |
| Use ∞-Driver instead of DER | Cross-city incremental learning, buffer 6,000 · Cross-city incremental learning, buffer 6,000 | Intransigence (I−1): -0.48 -> -0.28 | Hurt |
| Use ∞-Driver instead of PuriDivER | Cross-city incremental learning, buffer 9,000 · Cross-city incremental learning, buffer 9,000 | Average Loss (L−1): 0.72 -> 0.60 | Helped |
| Use ∞-Driver instead of PuriDivER | Cross-city incremental learning, buffer 9,000 · Cross-city incremental learning, buffer 9,000 | Forgetting (F−1): -0.08 -> -0.03 | Helped |
| Use ∞-Driver instead of DER | Cross-city incremental learning, buffer 9,000 · Cross-city incremental learning, buffer 9,000 | Intransigence (I−1): -0.38 -> -0.30 | Hurt |
| Use ∞-Driver instead of DER | Open-web incremental learning on 10 million YouTube images · Open-web incremental learning, 10 million YouTube images | Revised Average Loss (L̄−1): 1.24 -> 1.17 | Helped |
| Use ∞-Driver instead of DER | Open-web incremental learning on 10 million YouTube images · Open-web incremental learning, 10 million YouTube images | Revised Forgetting (F̄−1): -0.02 -> -0.02 | No clear effect |
| Use ∞-Driver instead of DER | Open-web-trained driving policy · Open-web evaluation set | ADE: 1.18 -> 1.13 | Helped |
Limitations
- Evaluation is primarily open-loop ADE; closed-loop CARLA results appear only in supplementary material.
- Relies on a supervised seed set (nuScenes-Boston) for the inverse dynamics teacher.
- Noisy pseudo-labels from ensemble disagreement still require heuristic filtering thresholds (three-sigma, GMM).
Experimental setup
Datasets: nuScenes KITTI Argoverse2 YouTube driving videos
Benchmarks: nuScenes KITTI Argoverse2 open web YouTube evaluation set
Metrics: average displacement error average loss forgetting intransigence
Baselines: DER Rainbow Memory PuriDivER
Builds on: conditional imitation learning Dark Experience Replay Rainbow Memory PuriDivER
Closest papers by meaning
- Learning to Drive is a Free Gift: Large-Scale Label-Free Autonomy Pretraining from Unposed In-The-Wild Videos (CVPR 2026 · similarity 0.89)
- Prioritizing Perception-Guided Self-Supervision: A New Paradigm for Causal Modeling in End-to-End Autonomous Driving (NeurIPS 2025 · similarity 0.88)
- Semi-supervised Class-Agnostic Motion Prediction with Pseudo Label Regeneration and BEVMix (AAAI 2024 · similarity 0.86)
- Knowledge Distillation from Single-Task Teachers to Multi-Task Student for End-to-End Autonomous Driving (AAAI 2024 · similarity 0.86)
- Panacea: Panoramic and Controllable Video Generation for Autonomous Driving (CVPR 2024 · similarity 0.86)
- Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning (AAAI 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.