Research map / Video / Video-language understanding
MUSE: Mamba Is Efficient Multi-scale Learner for Text-video Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MUSE, which builds a feature pyramid over CLIP's last single-scale feature map and models cross-resolution tokens with a gated residual Mamba (ResMamba) for text-video retrieval, reporting state-of-the-art results on MSR-VTT, DiDeMo and ActivityNet.
Problem
Text-video retrieval with plain CLIP-based encoders ignores multi-scale (cross-resolution) representations, and modeling those jointly with attention is quadratic in sequence length and memory-heavy.
Approach
Generate multi-scale features by pooling and convolving the last single-scale CLIP feature map (no extra backbone), then jointly model scale-wise token sequences with a linear-complexity bidirectional Mamba wrapped in a zero-initialized gated residual block (ResMamba).
Key results
- Plugged into T-MASS, MUSE reaches 50.9 R@1 text-to-video on MSR-VTT (vs 50.0 for T-MASS), and it improves CLIP4Clip from 42.6 to 44.8 R@1.
- MUSE achieves 49.7 video-to-text R@1 on MSR-VTT and 46.2 R@1 on ActivityNet, the best reported in the comparison tables.
- Replacing ResMamba with a Transformer raises training GPU memory from 12.6GB to 36.8GB per GPU, while Mamba gives the best MSR-VTT R@1 among the compared correlation modules (44.8 vs 43.0 for Transformer).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MUSE to T-MASS | T-MASS · MSR-VTT, text-to-video | R@1: 50.0 -> 50.9 (+0.9) | Helped |
| Add MUSE to CLIP4Clip | CLIP4Clip · MSR-VTT, text-to-video | R@1: 42.6 -> 44.8 (+2.2) | Helped |
| Replace Mamba with Transformer (ablation) | MUSE video learner · MSR-VTT | R@1: 44.8 -> 43.0 | Hurt |
| Replace Mamba with FlashAttention (ablation) | MUSE video learner · MSR-VTT | R@1: 44.8 -> 42.6 | Hurt |
| Replace Mamba with MambaOut, removing the SSM module (ablation) | MUSE video learner · MSR-VTT | R@1: 44.8 -> 42.4 | Hurt |
| Use bidirectional scan with separate projection weights (v2) instead of one-direction scan (ablation) | Mamba · MSR-VTT | R@1: 44.1 -> 44.8 | Helped |
| Aggregate features scale-wise instead of frame-wise or spatial-wise (ablation) | MUSE · MSR-VTT | R@1: 43.5 / 44.4 -> 44.8 | Helped |
| Add scale 28 to scales {1, 3, 7, 14} (ablation) | MUSE · MSR-VTT | R@1: 44.8 -> 42.5 | Hurt |
| Remove the ResMamba residual connection (ablation) | MUSE · MSR-VTT | R@1: 44.8 -> 43.9 | Hurt |
| Increase ResMamba layers from 4 to 16 (ablation) | MUSE · MSR-VTT | R@1: 44.8 -> 45.6 | Helped |
Surprising findings
- Adding MUSE at scale {1} only (no multi-scale) to CLIP4Clip still improves text-to-video R@1, which the authors attribute to Mamba itself.
- Scaling up to 4x resolution (scale 28) reduces R@1 (42.5 vs 44.8), attributed to redundancy from longer tokens.
Limitations
- Gains on the strongest baselines are modest (e.g., +0.9 R@1 on T-MASS text-to-video MSR-VTT), and results are reported with a single run per setting.
- Frame and scale settings were tuned on MSR-VTT, and most ablations use only MSR-VTT.
- Scale-wise aggregation and some design choices are chosen empirically without theoretical justification.
Experimental setup
Datasets: MSR-VTT ActivityNet DiDeMo
Benchmarks: MSR-VTT DiDeMo ActivityNet
Metrics: R@1 R@5 R@10 MdR MnR
Baselines: CLIP4Clip EMCL-Net STAN T-MASS X-Pool HBI DiffusionRet X-CLIP DiCoSA
Builds on: CLIP CLIP4Clip ViTDet Vim Mamba
Closest papers by meaning
- Towards Efficient and Effective Text-to-Video Retrieval with Coarse-to-Fine Visual Representation Learning (AAAI 2024 · similarity 0.87)
- Mitigating Semantic Collapse in Partially Relevant Video Retrieval (NeurIPS 2025 · similarity 0.87)
- TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignment (NeurIPS 2024 · similarity 0.87)
- Text Proxy: Decomposing Retrieval from a 1-to-N Relationship into N 1-to-1 Relationships for Text-Video Retrieval (AAAI 2025 · similarity 0.87)
- AdaVideoRAG: Omni-Contextual Adaptive Retrieval-Augmented Efficient Long Video Understanding (NeurIPS 2025 · similarity 0.86)
- Bridging the Semantic Granularity Gap Between Text and Frame Representations for Partially Relevant Video Retrieval (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.