Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Conformalized Interactive Imitation Learning: Handling Expert Shift and Intermittent Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends online quantile tracking to intermittently observed labels (IQT) and uses the resulting calibrated prediction intervals to drive ConformalDAgger, an interactive imitation learning method that queries experts when uncertainty rises under expert shift.
Problem
Uncertainty quantification for black-box imitation learning policies during deployment is overconfident under distribution shift, and expert labels in interactive IL arrive only intermittently, so standard online conformal methods do not apply directly.
Approach
Intermittent quantile tracking rescales the online quantile update by the probability of observing a label (err_t - alpha)/p_t times obs_t, retaining asymptotic coverage under intermittent feedback; ConformalDAgger uses the resulting interval width as a robot-gated query signal in DAgger.
Key results
- On Amazon stock with p_t=0.1, IQT with p_t-independent step size (lr small) boosts coverage toward the 0.9 target, while p_t-dependent variant gives larger intervals when lr is large.
- In simulated expert-shift experiments, ConformalDAgger raised the intervention rate from ~20% to ~60% at the shift episode, versus ~30% for EnsembleDAgger, with miscoverage peaking at 0.4 vs 1.0 for the baseline.
- On hardware with a 7DOF diffusion policy, ConformalDAgger queried more under the zig-zag shifted expert than under a stationary expert, with uncalibrated ensemble and safety baselines requesting fewer interventions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ConformalDAgger instead of EnsembleDAgger during expert shift | Interactive imitation learning · Simulated expert shift | intervention percentage: ~20% before shift -> ~60% in the 5th episode; EnsembleDAgger remains close to ~30% | Helped |
| use ConformalDAgger instead of EnsembleDAgger during expert shift | Interactive imitation learning · Simulated expert shift | miscoverage rate: max = 1.0 at shift, converges to 0.4 -> max = 0.4 at shift, converges to 0.2 | Helped |
| use ConformalDAgger rather than EnsembleDAgger under expert drift | Interactive imitation learning · Simulated expert drift | miscoverage rate | Helped |
| use ConformalDAgger rather than EnsembleDAgger with a stationary expert | Interactive imitation learning · Simulated stationary expert | alignment to expert | No clear effect |
| use ConformalDAgger rather than baselines under environment shift | Interactive imitation learning · Simulated environment shift | intervention frequency and policy learning quality | No clear effect |
| use ConformalDAgger rather than EnsembleDAgger or SafeDAgger with a shifted wiping strategy | Diffusion Policy on a 7DOF robotic manipulator · Hardware sponging task, shifted expert | robot-requested human feedback: about 35 timesteps | Helped |
| use ConformalDAgger rather than EnsembleDAgger or SafeDAgger with a shifted wiping strategy | Diffusion Policy on a 7DOF robotic manipulator · Hardware sponging task, shifted expert | learning the shifted strategy | Helped |
| use a p-dependent IQT update instead of a p-independent update (ablation) | IQT · Amazon stock prices, AR, infrequent observations | prediction interval size | Mixed |
| use a p-dependent IQT update instead of a p-independent update at smaller and midsized learning rates (ablation) | IQT · Amazon stock prices, AR, infrequent observations | coverage | No clear effect |
| use a step size that contains the recent maximum nonconformity score instead of a constant step size (ablation) | IQT · Standard time-series conformal prediction datasets | empirical coverage | Helped |
| lower the ConformalDAgger uncertainty threshold to 0.03 (ablation) | ConformalDAgger · Simulated expert shift, first deployment episode | intervention percentage: 70-80% | Hurt |
| raise the EnsembleDAgger uncertainty threshold to 0.09 (ablation) | EnsembleDAgger · Simulated expert shift | shift detection | Hurt |
Surprising findings
- Ensemble disagreement baselines remained overconfident and asked fewer questions when the expert shifted, while the conformal uncertainty rose immediately.
- Setting the step size proportional to p_t (p_t-dependent update) makes intervals excessively large when the learning rate is high.
Limitations
- Asymptotic coverage guarantees may not hold in finite-horizon robot deployments, as the authors acknowledge.
- Hardware experiments use a single human teleoperator with simulated 20% intervention probability and a hard threshold on interval size.
- Hardware evaluation covers a single task (sponge wiping) and limited deployment episodes.
Experimental setup
Datasets: Amazon stock prices Google stock prices Elec2 simulated goal-reaching demonstrations hardware teleoperated sponge-wiping demonstrations
Benchmarks: Amazon stock prices Google stock prices Elec2 simulated 4D cup-moving goal-reaching task 7DOF hardware sponge-wiping task
Metrics: marginal coverage miscoverage rate mean prediction interval size intervention percentage decision deviation trajectory deviation
Baselines: EnsembleDAgger LazyDAgger SafeDAgger implicit ensemble over diffusion samples
Builds on: DAgger online conformal prediction quantile tracking adaptive conformal inference EnsembleDAgger
Closest papers by meaning
- Uncertainty-Based Smooth Policy Regularisation for Reinforcement Learning with Few Demonstrations (NeurIPS 2025 · similarity 0.88)
- CCIL: Continuity-Based Data Augmentation for Corrective Imitation Learning (ICLR 2024 · similarity 0.87)
- Markov Balance Satisfaction Improves Performance in Strictly Batch Offline Imitation Learning (AAAI 2025 · similarity 0.87)
- Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloning (NeurIPS 2025 · similarity 0.87)
- Mitigating Covariate Shift in Behavioral Cloning via Robust Stationary Distribution Correction (NeurIPS 2024 · similarity 0.86)
- When Does Interaction Help? Representational Tradeoffs in Value-Based Imitation Learning (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.