Research map / Video / Action recognition
MANTA: Diffusion Mamba for Efficient and Effective Stochastic Long-Term Dense Action Anticipation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MANTA, a Mamba-based bidirectional selective state-space diffusion generator for stochastic long-term dense action anticipation, achieving SOTA on Breakfast, 50Salads, and Assembly101 with large speedups over GTDA.
Problem
Stochastic long-term dense action anticipation requires predicting future actions and their durations minutes ahead from video observations, while modelling uncertainty over both observed and future actions efficiently.
Approach
Replaces the dilated gated temporal convolutions of the GTDA diffusion generator with bidirectional selective state-space (Mamba-style) layers within a unified diffusion process, giving a global receptive field with linear complexity and data-dependent gating that distinguishes observed from masked future frames, removing the need for a multi-stage architecture.
Key results
- On Breakfast, MANTA improves Mean MoC over GTDA across all horizons (e.g., 34.2 vs 29.1 at alpha=0.3, beta=0.1).
- Inference is 65.3x faster than GTDA (1.1 s vs 71.8 s per video for S=25 samples) and training is 6.6x faster per epoch (57 s vs 379 s).
- MANTA has 2.8x fewer parameters than GTDA while achieving state-of-the-art results on all three datasets for both metrics (with some exceptions on Top-1 for specific settings).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MANTA instead of GTDA | Long-term dense action anticipation · Breakfast | Mean MoC: 24.0 -> 27.7 | Helped |
| Use MANTA instead of GTDA | Long-term dense action anticipation · Assembly101 | Top-1 MoC: 18.0 -> 16.9 | Hurt |
| Use MANTA instead of GTDA | Long-term dense action anticipation · 50Salads (α=0.3) | Top-1 MoC: 66.2 -> 71.7; 31.0 -> 31.1 | Mixed |
| Add selective, data-dependent gating to the BSSL layer (ablation) | MANTA · Breakfast (α=0.2, β=0.1) | Mean MoC: 26.4 -> 27.7 | Helped |
| Add selective, data-dependent gating to the BSSL layer (ablation) | MANTA · Breakfast (α=0.3, β=0.1) | Mean MoC: 31.7 -> 34.2 | Helped |
| Add selective, data-dependent gating to the BSSL layer (ablation) | MANTA · Breakfast (α=0.2, β=0.1) | Top-1 MoC: 47.3 -> 55.5 | Helped |
| Add selective, data-dependent gating to the BSSL layer (ablation) | MANTA · Breakfast (α=0.3, β=0.1) | Top-1 MoC: 51.6 -> 59.6 | Helped |
| Replace GTDA with MANTA | Breakfast inference generating 25 samples (α=0.3, β=0.5) · Breakfast | inference time: 71.8 seconds -> 1.1 seconds; 65.3× speedup | Helped |
| Replace GTDA with MANTA | Breakfast training · Breakfast | epoch runtime: 379 seconds -> 57 seconds | Helped |
| Replace GTDA with MANTA | Model · Not benchmark-specific | number of parameters: 2.8× fewer parameters | Helped |
Surprising findings
- Removing selectivity from the BSSL layer drops Top-1 MoC on Breakfast substantially (e.g., 55.5 to 47.3 at alpha=0.2, beta=0.1).
Limitations
- Does not outperform GTDA on every Top-1 setting (e.g., Assembly101 beta=0.1, 50Salads alpha=0.2).
Experimental setup
Datasets: Breakfast 50Salads Assembly101
Benchmarks: Breakfast 50Salads Assembly101
Metrics: Mean MoC Top-1 MoC
Baselines: Tri-gram UAAA DiffAnt GTDA
Builds on: GTDA Mamba DDIM DiffAnt S4
Closest papers by meaning
- Towards Understanding Future: Consistency Guided Probabilistic Modeling for Action Anticipation (AAAI 2024 · similarity 0.86)
- AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos? (ICLR 2024 · similarity 0.86)
- Stochastic Human Motion Prediction with Memory of Action Transition and Action Characteristic (CVPR 2025 · similarity 0.85)
- VEDIT: Latent Prediction Architecture For Procedural Video Representation Learning (ICLR 2025 · similarity 0.84)
- Can't Make an Omelette Without Breaking Some Eggs: Plausible Action Anticipation Using Large Video-Language Models (CVPR 2024 · similarity 0.84)
- E-Motion: Future Motion Simulation via Event Sequence Diffusion (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.