Research map / Reinforcement learning / Visual and representation RL

Novelty Detection in Reinforcement Learning with World Models

ICML 2025 · Reinforcement learning › Visual and representation RL · approach: In-context and adaptive RL · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a hyperparameter-free KL-divergence bound on DreamerV2 latent predictions to detect inference-time novelties in world-model RL agents, outperforming RIQN and a reconstruction-error baseline (PP-Mare) on MiniGrid, Atari, and DMC.

Problem

Detecting sudden, permanent changes in observation or transition dynamics (novelties) at inference time in RL agents, without manually tuned thresholds or augmented data.

Approach

A novelty-detection bound comparing KL divergences between the RSSM posterior conditioned on history and observation versus history-only or observation-only priors; a violation of the inequality flags novelty, requiring no thresholds. Also a reconstruction-error ablation (PP-Mare) with a tuned threshold.

Key results

• KL bound achieves false positive rate <=10^-2 across all tested Atari and DMC nominal environments, versus 0.17-0.43 for RIQN on Atari.
• KL bound gives inference speedup of about 1.34x10^3 over RIQN on DMC and 5.12x10^2 on Atari.
• On Atari Freeway InvisibleCars, ColorCars and FrozenCars, the KL bound reaches ADE <=10^-2 with AUC >=0.99 on the first two.

Measured effects reported in the paper

Surprising findings

• RIQN achieves lower delay error on several DMC novelties, but at false positive rates as high as 68%.
• The policy-restricted exploration of the nominal policy can make unvisited but nominal transitions appear novel, a failure mode of world-model-based detection.

Limitations

• KL divergence is not a metric, so the bound's guarantee is approximate and may be violated for reasons other than novelty.
• Exploration effects from the learned policy can cause nominal but rarely visited transitions to be flagged as novel.
• The KL bound cannot be applied directly to diffusion-based world models that generate images rather than latent states.

Experimental setup

Datasets: MiniGrid Atari DeepMind Control Suite NovGrid HackAtari RealWorldRL Suite

Benchmarks: MiniGrid novelties (DoorKey-6x6 variants) Atari Freeway, Kangaroo, SeaQuest, Boxing novelties DMC Cartpole, Quadruped, Humanoid, Walker novelties

Metrics: average delay error AUC false positive rate inference speedup

Baselines: RIQN PP-Mare

Builds on: DreamerV2 RIQN Bayesian surprise CUSUM

Closest papers by meaning

• Beyond Prediction: Steering VLM Agents with Retrospective World Modeling (NeurIPS 2026 · similarity 0.87)
• RLVR-World: Training World Models with Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
• Open-World Reinforcement Learning over Long Short-Term Imagination (ICLR 2025 · similarity 0.86)
• Simplifying Latent Dynamics with Softly State-Invariant World Models (NeurIPS 2024 · similarity 0.86)
• Learning World Models for Unconstrained Goal Navigation (NeurIPS 2024 · similarity 0.86)
• Novelty-Guided Data Reuse for Efficient and Diversified Multi-Agent Reinforcement Learning (AAAI 2025 · similarity 0.86)