Research map / Video / Action recognition
Towards Understanding Future: Consistency Guided Probabilistic Modeling for Action Anticipation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a Consistency-guided Probabilistic Model (CPM) on a Transformer that uses variational (ELBO-style) probabilistic modeling and global semantics consistency losses to smooth past-to-future temporal consistency for egocentric action anticipation.
Problem
Action anticipation predicts future actions from past video segments, but existing methods ignore temporal continuity and semantic consistency between past and future segments, leading to inconsistent or unexpected anticipations.
Approach
Adds three modules to a Transformer during training: probabilistic future semantics estimation (Gaussian past-aware feature), a global semantics loss aligning past features with past-and-future features, and a global distribution module that minimizes KL distance between past-aware and global-aware Gaussians, motivated by an ELBO over future features. Only the future semantics module is used at te
Key results
- On EK100 validation, CPM with RGB reaches 17.2 Recall@5 versus 16.1 for DCR-1s (TSM); with RGB+OBJ+Flow it reaches 19.4 vs 18.3 for DCR-1s.
- On EK55, CPM achieves 17.2 Top-1 and 35.2 Top-5 (RGB) and 20.1 Top-1 and 43.6 Top-5 (multi-modal), outperforming DCR-1s.
- On EGTEA Gaze+, CPM achieves 50.3 Top-1 and 39.1 CM Top-1, above Abstract Goal (49.8 / 37.4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CPM consistency-guided probabilistic modeling | TSM-based action anticipation · EK100 validation, RGB | action Recall@5: 16.6 -> 17.2 | Helped |
| Add CPM consistency-guided probabilistic modeling | TSM-based action anticipation · EK100 validation, RGB+OBJ+Flow | action Recall@5: 18.3 -> 19.4 | Helped |
| Add CPM consistency-guided probabilistic modeling | TSM-based action anticipation · EK55 validation, RGB | Top-1 accuracy: 16.1 -> 17.2 | Helped |
| Add CPM consistency-guided probabilistic modeling | TSM-based action anticipation · EK55 validation, RGB | Top-5 accuracy: 33.1 -> 35.2 | Helped |
| Add CPM consistency-guided probabilistic modeling | TSM-based action anticipation · EK55 validation, RGB+OBJ+Flow | Top-1 accuracy: 19.2 -> 20.1 | Helped |
| Add CPM consistency-guided probabilistic modeling | TSM-based action anticipation · EK55 validation, RGB+OBJ+Flow | Top-5 accuracy: 41.2 -> 43.6 | Helped |
| Add CPM consistency-guided probabilistic modeling | TSM-based action anticipation · EGTEA-GAZE+ split 1 | Top-1 accuracy: 49.8 -> 50.3 | Helped |
| Add CPM consistency-guided probabilistic modeling | TSM-based action anticipation · EGTEA-GAZE+ split 1 | CM Top-1 accuracy: 37.4 -> 39.1 | Helped |
| Use a 3-layer encoder and 4-layer future decoder (ablation) | CPM · EK100 validation | verb Top-1 accuracy: 34.5 -> 36.9 | Helped |
| Use a 4-layer encoder and 4-layer future decoder instead of a 3-layer encoder and 4-layer decoder (ablation) | CPM · EK100 validation | verb Top-1 accuracy: 36.9 -> 33.1 | Hurt |
| Use negative Jaccard distance instead of negative dot, L2, or negative cosine distance in Lsem and Lrec (ablation) | CPM · EK100 validation | verb Top-1 accuracy: 33.1; 34.0; 34.3 -> 36.9 | Helped |
| Add Lsem to the base action loss (ablation) | CPM · EK100 validation | verb Top-1 accuracy: 32.1 -> 33.5 | Helped |
Limitations
- Gains over strongest prior methods are modest (e.g., about 0.5 Top-1 on EG+ over Abstract Goal).
- Smoothness is evaluated mainly on EK100 with feature-curve metrics rather than downstream effects alone.
Experimental setup
Datasets: EpicKitchens-100 EpicKitchens-55 EGTEA Gaze+
Benchmarks: EpicKitchens-100 EpicKitchens-55 EGTEA Gaze+
Metrics: Recall@5 Top-1 Top-5 class-mean Top-1 MATD mean curvature
Baselines: DCR-1s (TSN) DCR-1s (TSM) AVT MeMViT-16 RULSTM ActionBanks HRO AVT++ TransAction Ego-OMG Abstract Goal I3D-Res50
Builds on: Transformer variational autoencoder DCR Abstract Goal TSM
Closest papers by meaning
- Can't Make an Omelette Without Breaking Some Eggs: Plausible Action Anticipation Using Large Video-Language Models (CVPR 2024 · similarity 0.87)
- AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos? (ICLR 2024 · similarity 0.87)
- MANTA: Diffusion Mamba for Efficient and Effective Stochastic Long-Term Dense Action Anticipation (CVPR 2025 · similarity 0.86)
- E-Motion: Future Motion Simulation via Event Sequence Diffusion (NeurIPS 2024 · similarity 0.86)
- PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning (ICML 2025 · similarity 0.86)
- Video-Mirai: Autoregressive Video Diffusion Models Need Foresight (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.