Research map / Video / Action recognition
LookWhen? Fast Video Recognition by Learning When, Where, and What to Compute
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LookWhen, a shallow-selector plus deep-extractor video model pre-trained to select unique tokens (top1-distance on DINOv3 features) and distill InternVideo2 and time-normalized DINOv3 targets, achieving better accuracy-FLOPs and accuracy-throughput trade-offs for video recognition.
Problem
Video transformers split videos into many tokens with superlinear compute cost, and existing adaptive token reduction methods either reduce FLOPs without real speedups or lack general, positionally-grounded selection signals for video.
Approach
Extends the LookWhere selector-extractor to video: a shallow selector on a downscaled video scores tokens by feature-space uniqueness (distance to nearest neighbor in DINOv3 features), and a full-depth extractor processes only the top-K tokens while distilling InternVideo2 video tokens and time-normalized DINOv3 frame/patch/video tokens.
Key results
- LookWhen reaches 84.6% on K400 at 70% sparsity, 1.1% below UMT-B800e while using 40% fewer FLOPs, and 0.6% above VideoMambaPro at 78% fewer FLOPs.
- On SSv2, LookWhen scores 72.0% at 108 GFLOPs per view versus VideoMAEv2's 71.2% at 180 GFLOPs.
- LookWhen Pareto-dominates the upgraded baselines on 9 of 12 accuracy-FLOPs cases and roughly matches on the other 3; at equal accuracy it is 6.7x faster than InternVideo2-B on an L40S GPU.
Surprising findings
- InternVideo2 attention-based and feature-based selection targets perform no better than random selection, while DINOv3-based uniqueness targets work well.
- LookWhen sometimes outperforms its own dense video teacher, InternVideo2.
Limitations
- Restricted to ViT-Base scale due to compute constraints; larger ViTs not tested.
- Requires a teacher with positionally-grounded patch-level features to supervise the selector.
Experimental setup
Datasets: Kinetics-400 Something-Something-v2 Epic-Kitchens-100 Diving48 Jester Charades
Benchmarks: Kinetics-400 Something-Something-v2 Epic-Kitchens-100 Diving48 Jester Charades
Metrics: top-1 accuracy GFLOPs throughput peak memory
Baselines: InternVideo2-B vid-TLDR RLT LITE ToMe STTS UMT-B800e VideoMambaPro VideoMAEv2
Builds on: LookWhere InternVideo2 DINOv3
Closest papers by meaning
- LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervision (NeurIPS 2025 · similarity 0.90)
- FrameExit: Conditional Early Exiting for Efficient Video Recognition (CVPR 2021 · similarity 0.87)
- LookThere! Sparse Vision by Reinforced Selection (NeurIPS 2026 · similarity 0.86)
- FlexSelect: Flexible Token Selection for Efficient Long Video Understanding (NeurIPS 2025 · similarity 0.86)
- No More Shortcuts: Realizing the Potential of Temporal Self-Supervision (AAAI 2024 · similarity 0.85)
- Taylor Videos for Action Recognition (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.