Research map / Video / Video segmentation and tracking
MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoMa, a PEFT adapter that couples window-based local attention (Divide) with a Mamba-derived sequence modulation operation SeqMod (Modulate) to adapt frozen CLIP ViTs to video recognition.
Problem
Adapting image foundation models to video with parameter-efficient fine-tuning usually handles spatial and temporal information separately, so patches interact only indirectly, while full spatial-temporal attention is quadratic in sequence length.
Approach
Insert a lightweight Mamba SSM into each frozen CLIP layer that outputs per-token scale and bias sequences; these modulate the attention output via a sequence-level, residual-preserving SeqMod operation, after window-based local attention (Divide) has been applied.
Key results
- On Kinetics-400 with ViT-L/14 at 16 frames, MoMa reaches 87.8 top-1 with 39M tunable parameters, versus 87.6 for DiST and 87.3 for AIM.
- On Something-Something-v2 with ViT-L/14 at 32 frames, MoMa reports 73.8 top-1, above DiST's 73.1.
- In the SeqMod ablation on K400 with ViT-B/16, SeqMod reaches 84.8 top-1, above Raw-AdaN (78.9) and Skip (72.4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace scalar AdaN scale and bias with sequence-wise SeqMod scale and bias (ablation) | MoMa sequence fusion · Kinetics-400 | Top-1: 78.9 -> 84.8 | Helped |
| Fuse the sequences by 1:1 addition instead of skipping the SSM block (ablation) | MoMa sequence fusion · Kinetics-400 | Top-1: 72.4 -> 69.3 | Hurt |
| Fuse the sequences by element-wise maximum instead of skipping the SSM block (ablation) | MoMa sequence fusion · Kinetics-400 | Top-1: 72.4 -> 70.2 | Hurt |
| Concatenate the sequences and use a linear layer rather than skipping the SSM block (ablation) | MoMa sequence fusion · Kinetics-400 | Top-1: 72.4 -> 75.5 | Helped |
| Use full attention instead of 8×8 windowed attention (ablation) | MoMa Divide stage · SSv2 (640×480×16 input) | Top-1: 70.1 -> 69.2 | Hurt |
| Increase Divide window size from 8×8 to 16×16 (ablation) | MoMa Divide stage · SSv2 (640×480×16 input) | Top-1: 70.1 -> 70.3 | Helped |
| Reduce Divide window size from 8×8 to 4×4 (ablation) | MoMa Divide stage · SSv2 (640×480×16 input) | Top-1: 70.1 -> 67.5 | Hurt |
| Use a 3D 4×4×4 Divide window instead of an 8×8 2D window (ablation) | MoMa Divide stage · SSv2 (640×480×16 input) | Top-1: 70.1 -> 68.2 | Hurt |
| Alternate Transformer and SSM layers uniformly rather than placing all Transformer layers before SSM layers (ablation) | MoMa layer design · Kinetics-400 | Top-1: 80.6 -> 83.7 | Helped |
| Use uniform [TM]12 alternation rather than the [T]6[TMM]6 pattern (ablation) | MoMa layer design · Kinetics-400 | Top-1: 81.5 -> 83.7 | Helped |
| Use uniform [TM]12 alternation rather than the [TTMM]6 pattern (ablation) | MoMa layer design · Kinetics-400 | Top-1: 82.8 -> 83.7 | Helped |
| Use MoMa rather than DiST for zero-shot recognition | CLIP ViT-B/16 · HMDB51 | accuracy: 55.4 -> 56.2 | Helped |
Surprising findings
- Simple fusions (Add and Max) of the SSM output perform worse than skipping the SSM block entirely (69.3 and 70.2 vs 72.4 top-1).
- Full-image attention in the Divide stage performs worse than 8x8 windowed attention on SSv2 (69.2 vs 70.1 top-1), which the authors attribute to CLIP being trained on attention over short sequences.
Limitations
- Ablations are run on K400 with ViT-B/16 only, and the CLIP distillation loss and training details are only lightly described.
- Zero-shot results are limited to HMDB51 and UCF101, and the claimed gains over DiST are modest (56.2 vs 55.4 on B/16).
Experimental setup
Datasets: Kinetics-400 Something-Something-v2 Breakfast COIN
Benchmarks: Kinetics-400 Something-Something-v2 Breakfast COIN HMDB51 UCF101
Metrics: Top-1 accuracy Top-5 accuracy GFLOPs tunable parameters frames per second GPU memory
Baselines: AIM DiST EVL UMT ActionCLIP X-CLIP VideoMamba ViS4mer Turbo MViT VideoSwin
Builds on: Mamba CLIP AIM DiST VideoMamba FiLM
Closest papers by meaning
- Mamba-Adaptor: State Space Model Adaptor for Visual Recognition (CVPR 2025 · similarity 0.89)
- Spatial-Mamba: Effective Visual State Space Models via Structure-Aware State Fusion (ICLR 2025 · similarity 0.86)
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (CVPR 2024 · similarity 0.86)
- Vision Mamba Mender (NeurIPS 2024 · similarity 0.86)
- Where Do We (Not) Need Temporal Context in Low-Resource Video Task Adaptation? (NeurIPS 2026 · similarity 0.86)
- H-MBA: Hierarchical MamBa Adaptation for Multi-Modal Video Understanding in Autonomous Driving (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.