Research map / Video / Video segmentation and tracking
Token Bottleneck: One Token to Remember Dynamics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Token Bottleneck (ToBo), a self-supervised pretraining pipeline that squeezes a reference frame into a single CLS bottleneck token and reconstructs a heavily masked target frame from that token plus few target patches, yielding backbones for sequential scene understanding.
Problem
Learning compact yet temporally aware visual representations from dynamic scenes for sequential tasks such as visual tracking, label propagation, and robotic manipulation, where static-image SSL and patch-correspondence methods (e.g., SiamMAE, RSP) fall short.
Approach
Encode a reference frame into a bottleneck CLS token and reconstruct a target frame with a very high masking ratio (0.9) using only that token and few visible target patches, forcing the token to summarize the scene while embedding temporal dynamics; uses a self-attention-only decoder rather than cross-attention.
Key results
- On Franka Kitchen (ViT-S/16, Kinetics-400), ToBo gains over the second-best baseline on all five tasks, e.g., Knob1 on 57.0 vs 31.5 and Ldoor open 51.0 vs 28.8.
- On CortexBench, ToBo reaches 73.5 on DMC vs 61.6 for RSP (+11.9) and 60.4 on Adroit vs 50.0 for CropMAE (+10.4).
- On DAVIS J&F mean, ToBo scores 60.6 vs 60.1 for RSP; on JHMDB PCK@0.1 it scores 47.0 vs 44.7 for SiamMAE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ToBo instead of the strongest listed baseline | ViT-S/16 robot policy representations · Franka Kitchen tasks | success rate (%): 57.0{ $\pm$2.0} vs \underline{31.5}{ $\pm$5.3}; 82.0{ $\pm$1.6} vs \underline{55.8}{ $\pm$6.4}; 95.0{ $\pm$7.1} vs \unde | Helped |
| Use ToBo instead of the strongest listed baseline | ViT-S/16 robot policy representations · CortexBench tasks | success rate (%) / normalized score: 60.4{ $\pm$2.2} vs \underline{50.0}{ $\pm$5.1}; 87.8{ $\pm$4.6} vs \underline{84.5}{ $\pm$6.6}; 73.5{ $\pm$0.9} vs \unde | Helped |
| Use ToBo instead of the strongest listed baseline | ViT-S/16 robot policy representations · RLBench tasks | success rate (%): 41.2{ $\pm$7.4} vs \underline{28.4}{ $\pm$3.0}; 52.3{ $\pm$3.2} vs \underline{48.0}{ $\pm$4.6}; 42.2{ $\pm$6.9} vs \unde | Helped |
| Use ToBo instead of the strongest listed baseline | Kinetics-400-pretrained representations · DAVIS, VIP, and JHMDB video label propagation | J&F_m / J_m / F_m / mIoU / PCK: 60.6 vs 60.1; 58.4 vs 57.4; 63.0 vs 62.8; 34.0 vs 33.8; 47.0 vs 44.7; 74.8 vs 73.4 | Helped |
| Increase target-scene masking ratio up to 0.9 (ablation) | ToBo · Franka Kitchen | success rate (%) | Helped |
| Increase target-scene masking ratio to 0.95 (ablation) | ToBo · Franka Kitchen | success rate (%) | Hurt |
| Use 2, 4, or 8 bottleneck tokens instead of one (ablation) | ToBo · Franka Kitchen mean success rate (%) | success rate (%): 61.1 -> 41.8; 61.1 -> 36.1; 61.1 -> 24.3 | Hurt |
| Set maximum temporal difference to 48 rather than 96 (ablation) | ToBo · Franka Kitchen mean success rate (%) | success rate (%): 61.1 -> 58.9 | Hurt |
| Set maximum temporal difference to 144 rather than 96 (ablation) | ToBo · Franka Kitchen mean success rate (%) | success rate (%): 61.1 -> 57.7 | Hurt |
| Remove temporal difference by using the same source and target frame (ablation) | ToBo · Franka Kitchen mean success rate (%) | success rate (%): 61.1 -> 53.5 | Hurt |
| Use four source frames instead of one (ablation) | ToBo · Franka Kitchen mean success rate (%) | success rate (%): 61.1 -> 46.9 | Hurt |
Surprising findings
- Using a single bottleneck token outperformed 2, 4, and 8 tokens on Franka Kitchen, with performance dropping as token count grows.
- Training without any temporal difference (same frame as source and target) still outperformed MAE, SiamMAE, and RSP, though below the full temporal version.
Limitations
- Results on RLBench and some tasks show high variance across seeds; scaling beyond ViT-L not explored.
- Multi-frame extension performed worse than the two-frame ToBo, suggesting naive extension is challenging.
- Comparisons with vision-language and robot representation models use unbalanced training setups.
Experimental setup
Datasets: Kinetics-400 DAVIS VIP JHMDB Franka Kitchen RLBench CortexBench
Benchmarks: Franka Kitchen CortexBench RLBench DAVIS VIP JHMDB real-world UR5e manipulation tasks
Metrics: success rate (%) normalized score J&F mean mIoU PCK@0.1 PCK@0.2
Baselines: SimCLR MoCo v3 DINO MAE SiamMAE RSP CropMAE R3M MVP Voltron MPI Theia
Builds on: masked autoencoder SiamMAE CropMAE RSP
Closest papers by meaning
- Visual Representation Learning with Stochastic Frame Prediction (ICML 2024 · similarity 0.87)
- DynaTokens: Teaching Dynamics to Camera-Controlled Video Models at Test Time (NeurIPS 2026 · similarity 0.86)
- ODTrack: Online Dense Temporal Token Learning for Visual Tracking (AAAI 2024 · similarity 0.86)
- ROVER: Recursive Reasoning Over Videos with Vision-Language Models for Embodied Tasks (NeurIPS 2025 · similarity 0.85)
- DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation (NeurIPS 2026 · similarity 0.85)
- Less Is More: Token Context-Aware Learning for Object Tracking (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.