Research map / Video / Action recognition
Manta: Enhancing Mamba for Few-Shot Action Recognition of Long Sub-Sequence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Manta, a Mamba-2-based few-shot action recognition framework with nested Matryoshka Mamba modules for local feature modeling and implicit alignment, plus a hybrid supervised/unsupervised contrastive loss, for long video sub-sequences.
Problem
Few-shot action recognition with long video sub-sequences is hard: Transformers are too costly for long inputs, plain Mamba neglects local features and temporal alignment, and intra-class variance accumulates as sequences grow longer.
Approach
Nest multiple Inner Mamba-2 modules (for local fragments) inside a bidirectional Outer Mamba-2 module (for timeline alignment) across multiple scales, and add a hybrid contrastive loss (supervised on support, unsupervised on query and all samples) to reduce intra-class variance.
Key results
- With ResNet-50 on SSv2 1-shot (8 frames), Manta reaches 63.4% vs 61.7% for prior SOTA AMFAR.
- On Kinetics 5-shot with ViT-B, Manta reaches 96.3%, versus 95.8% for the best prior RGB-based MoLo (ViT-B).
- Under long sub-sequences (F=16, ResNet-50 ablation), full Manta reaches 64.7% SSv2 1-shot vs 46.3% for the baseline without components; inference on 10,000 tasks takes 4.25 h on SSv2 1-shot vs 8.67 h for TRX.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the Inner Module (ablation) | Manta · SSv2 1-shot | accuracy: 46.3 -> 55.5 | Helped |
| Add the Outer Module (ablation) | Manta · SSv2 1-shot | accuracy: 46.3 -> 55.3 | Helped |
| Add hybrid contrastive learning loss (ablation) | Manta · SSv2 1-shot | accuracy: 46.3 -> 48.0 | Helped |
| Use all three components: IM, OM, and hybrid contrastive loss (ablation) | Manta · SSv2 1-shot | accuracy: 63.8 -> 64.7 | Helped |
| Use multi-scale O={1,2,4} instead of O={1} (ablation) | Manta · SSv2 1-shot | accuracy: 63.3 -> 64.7 | Helped |
| Use four scales O={1,2,4,8} instead of three scales O={1,2,4} (ablation) | Manta · SSv2 1-shot | accuracy: 64.7 -> 64.1 | Hurt |
| Use Mamba-2 in both Inner and Outer Modules instead of Mamba-1 in both (ablation) | Manta · SSv2 1-shot | accuracy: 64.1 -> 64.7 | Helped |
| Use independent Inner Module parameters and shared Outer Module parameters (ablation) | Manta · SSv2 1-shot | accuracy: 63.2 -> 64.7 | Helped |
| Use learnable scale weights instead of fixed weights (ablation) | Manta · SSv2 1-shot | accuracy: 62.2 -> 64.7 | Helped |
| Reverse the support-set temporal order (ablation) | Manta · SSv2 1-shot | accuracy: 64.7 -> 58.2 | Hurt |
| Set the loss weight to lambda=4 instead of lambda=1 (ablation) | Manta · SSv2 1-shot | accuracy: 60.8 -> 64.7 | Helped |
| Use all three contrastive losses together instead of supervised contrastive loss alone (ablation) | Manta · SSv2 1-shot | accuracy: 63.3 -> 64.7 | Helped |
Surprising findings
- Reversing the support temporal order drops SSv2 1-shot accuracy from 64.7% to 58.2%, showing sensitivity to temporal order.
- Plain Mamba-based baseline without the proposed modules performs far worse than prior Transformer methods on SSv2 (46.3% 1-shot).
Limitations
- Experiments mainly with ResNet-50 at F=16; results for longer sub-sequences are only reported in the ablation curves.
- Relies on the Matryoshka scale hyper-parameters chosen on validation sets; more scales give no gain.
- Hybrid contrastive loss adds three terms and a weight factor lambda requiring tuning.
Experimental setup
Datasets: SSv2 Kinetics UCF101 HMDB51 ImageNet
Benchmarks: SSv2 Kinetics UCF101 HMDB51
Metrics: accuracy DTW score inference time
Baselines: STRM SloshNet SA-CT GCSM GgHM TRX HyRSM MoLo AMFAR MTFAN Lite-MKD
Builds on: Mamba-2 OTAM supervised contrastive learning VMamba
Closest papers by meaning
- TAMT: Temporal-Aware Model Tuning for Cross-Domain Few-Shot Action Recognition (CVPR 2025 · similarity 0.86)
- Frame Order Matters: A Temporal Sequence-Aware Model for Few-Shot Action Recognition (AAAI 2025 · similarity 0.85)
- Vision Mamba Mender (NeurIPS 2024 · similarity 0.85)
- Temporal Alignment-Free Video Matching for Few-shot Action Recognition (CVPR 2025 · similarity 0.84)
- MambaOut: Do We Really Need Mamba for Vision? (CVPR 2025 · similarity 0.83)
- Mamba Can Learn Low-Dimensional Targets In-Context via Test-Time Feature Learning (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.