Research map / Video / Action recognition

Smooth Regularization for Efficient Video Recognition

NeurIPS 2025 · Video › Action recognition · approach: Contrastive video representation learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a Gaussian Random Walk (GRW) smoothness regularizer over frame embeddings, combining a permutation-based contrastive frame-ordering loss with an acceleration prior, to improve lightweight video recognition models (MoViNets, MobileNetV3) on Kinetics.

Problem

Lightweight video recognition models under-exploit temporal structure; the paper seeks a temporal inductive bias that improves accuracy within tight FLOP and memory budgets.

Approach

Model frame-to-frame embedding velocities as a Gaussian random walk, so the loss penalizes large accelerations; a contrastive loss over permutations of frame order prevents collapse to a single point. Applied as an extra loss term on intermediate or final-layer embeddings with a small Transformer head.

Key results

• MoViNet-A0-S-GRW reaches 78.4% Top-1 on K600 at 2.7 GFLOPs vs 72.3% for MoViNet-A0, and MoViNet-A3-GRW reaches 85.6% at 56.4 GFLOPs, the claimed SOTA in its FLOP group.
• MobileNetV3-S-GRW reaches 67.3% Top-1 on K600 at 30 MB vs 61.3% baseline.
• Ablation on K600 (MoViNet-A2-S): final-layer GRW gives 83.3% vs 78.4% baseline and 79.3% for attention-only head.

Measured effects reported in the paper

Surprising findings

• Adding only the 2-layer Transformer head without GRW gives just +0.9 Top-1, so most gain is attributed to the smoothing loss.
• Empirically centering is not required when a learnable linear transformation precedes GRW.

Limitations

• Assumes Gaussian random walk dynamics, which may not hold for abrupt or discontinuous motion.
• Not evaluated on very large-capacity models due to compute constraints.
• Frame-ordering permutation loss adds complexity; denominator requires enumeration or sampling of permutations.

Experimental setup

Datasets: Kinetics-600 Kinetics-400 Airplanes (warm-up toy dataset, 1,000 train videos)

Benchmarks: Kinetics-600 Kinetics-400

Metrics: Top-1 accuracy GFLOPs memory (MB)

Baselines: MoViNet-A0 MoViNet-A1 MoViNet-A2 MoViNet-A3 MobileNetV3-S MobileNetV3-S+TSM X3D MViTv2-B-32x3 UniFormer VideoMamba

Builds on: MoViNets MobileNetV3 slow feature analysis TSM

Closest papers by meaning

• Slow and Steady Feature Analysis: Higher Order Temporal Coherence in Video (CVPR 2016 · similarity 0.87)
• DejaVid: Encoder-Agnostic Learned Temporal Matching for Video Classification (CVPR 2025 · similarity 0.85)
• GenRec: Unifying Video Generation and Recognition with Diffusion Models (NeurIPS 2024 · similarity 0.84)
• TTVidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining (NeurIPS 2026 · similarity 0.84)
• LookWhen? Fast Video Recognition by Learning When, Where, and What to Compute (NeurIPS 2026 · similarity 0.84)
• Instant Video Models: Universal Adapters for Stabilizing Image-Based Networks (NeurIPS 2025 · similarity 0.84)