Research map / Reinforcement learning / Visual and representation RL

Self-Supervised Keyframe Discovery for Horizon-Invariant Behavior Cloning

NeurIPS 2026 · Reinforcement learning › Visual and representation RL · approach: In-context and adaptive RL

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Keyframe Mnemonics learns a self-supervised keyframe-relevance reward from randomly masked past observations, trains a PPO selector to pick keyframes, and conditions BC policies on the selected mnemonics for horizon-invariant generalization.

Problem

Behavior cloning in non-Markovian, partially observable tasks needs long-horizon memory, but recurrent and attention-based policies suffer from state collapse and limited context length and fail to generalize beyond training horizons.

Approach

Train a proxy policy on randomly masked past-observation buffers so simplicity bias makes it rely on keyframes; use its loss as a reward for an RL selector that fills a priority-queue memory buffer with keyframes; then train BC policies on the mnemonic buffer plus current observation.

Key results

• Mnemonic-conditioned MLP policies reach 100% SR on synthetic and grid memory domains and generalize to horizons orders of magnitude beyond training.
• Across 23 MIKASA-Robo tasks, KM improves average absolute SR by 13.9% over ELMUR, outperforming it on 21 of 23 tasks.
• On the real robot, KM retains about 80% SR at 20x longer test horizons, while DP and PTP deteriorate.

Surprising findings

• ELMUR stays at 0% SR for delays beyond the training horizon even when trained for about 10 hours, suggesting extra training does not fix horizon generalization.

Limitations

• Memory budget k is hard to predetermine, and multiple information-equivalent buffer states can limit correct keyframe retention.
• Each demonstration must contain at least one keyframe, which limits multi-task and play-demonstration settings.
• Theoretical guarantees rely on strong assumptions (approximate belief sufficiency, simplicity-biased SGD approximation).

Experimental setup

Datasets: MIKASA-Robo default dataset Robomimic Add Scattered Copy Active T-Maze MiniGrid-Memory LTMB-Hallway

Benchmarks: Add Scattered Copy Active T-Maze MiniGrid-Memory LTMB-Hallway MIKASA-Robo RememberColor3-ΔT Robomimic (Lift, Can, Square) real-world RememberColor2

Metrics: success rate MSE active memory regret passive memory regret

Baselines: LSTM (BPTT/TBPTT) Decision-LSTM Decision Transformer Decision-Mamba Transformer-XL ELMUR PTP RATE Conservative Q-Learning Diffusion Policy

Builds on: Decision Transformer PPO ELMUR MIKASA-Robo Diffusion Policy

Closest papers by meaning

• Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning (NeurIPS 2024 · similarity 0.88)
• Rethinking Latent Redundancy in Behavior Cloning: An Information Bottleneck Approach for Robot Manipulation (ICML 2025 · similarity 0.88)
• Improving Generative Behavior Cloning via Self-Guidance and Adaptive Chunking (NeurIPS 2025 · similarity 0.88)
• Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representations (NeurIPS 2025 · similarity 0.87)
• Learning When to Stop: Selective Imitation Learning Under Arbitrary Dynamics Shift (NeurIPS 2026 · similarity 0.87)
• SkiP: When to Skip and When to Refine for Efficient Robot Manipulation (NeurIPS 2026 · similarity 0.87)