Research map / Video / Video-language understanding
Mirasol3B: A Multimodal Autoregressive Model for Time-Aligned and Contextual Modalities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Mirasol3B decouples multimodal modeling into an autoregressive model for time-aligned audio+video (with a Combiner that compresses snippets, including a Token Turing Machine variant) and a separate autoregressive text model, scaling to 512 frames with about 3B parameters.
Problem
Combining heterogeneous modalities (video, audio, and global text context) that differ in sampling rate and volume, while handling long video inputs efficiently.
Approach
Split autoregressive modeling into two components: one for time-synchronized audio-video chunks, compressed by a causal Combiner (Transformer or Token Turing Machine based) into a few tokens per chunk, and one for unaligned text that cross-attends to the audio-video latents.
Key results
- Mirasol3B reaches 50.42% accuracy on MSRVTT-QA, outperforming Flamingo (47.4%) and M-PLUG2 (48.0%) with fewer than 3B parameters.
- On NExT-QA, Mirasol3B with 512 frames and TTM Combiner reaches 73.2%, versus 63.1% for HiTeA.
- On Kinetics-Sound, Mirasol3B (A+V) reaches 90.1% versus 85.0% for MBT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Mirasol3B instead of Flamingo | Video question answering · MSRVTT-QA | accuracy: 47.4 -> 50.42 | Helped |
| Use Mirasol3B with 512 frames instead of 128 frames | Mirasol3B · ActivityNet | accuracy: 48.25 -> 51.13 | Helped |
| Use Mirasol3B with 512 frames instead of 128 frames | Mirasol3B · NExT-QA | accuracy: 68.2 -> 72.0 | Helped |
| Add audio to video input | Mirasol3B on Kinetics-Sound · Kinetics-Sound | accuracy: 81.3 -> 90.1 | Helped |
| Add autoregressive modeling (ablation) | MSRVTT-QA baseline · MSRVTT-QA | accuracy: 41.5 -> 43.2 | Helped |
| Add Combiner (ablation) | MSRVTT-QA baseline · MSRVTT-QA | accuracy: 41.5 -> 42.1 | Helped |
| Add pretraining (ablation) | MSRVTT-QA baseline · MSRVTT-QA | accuracy: 41.5 -> 45.2 | Helped |
| Combine autoregressive modeling, Combiner, and pretraining (ablation) | MSRVTT-QA baseline · MSRVTT-QA | accuracy: 41.5 -> 47.9 | Helped |
| Replace Perceiver Combiner with the proposed Transformer Combiner (ablation) | Mirasol3B · MSRVTT-QA | accuracy: 43.1 -> 44.2 | Helped |
| Replace Transformer Combiner with TTM Combiner (ablation) | Mirasol3B · MSRVTT-QA | accuracy: 44.2 -> 44.8 | Helped |
| Process frames in 8 autoregressive chunks rather than 1 chunk (ablation) | Mirasol3B · MSRVTT-QA | accuracy: 41.8 -> 45.1 | Helped |
| Replace autoregressive modeling with bidirectional modeling (ablation) | Mirasol3B · MSRVTT-QA | accuracy: 45.1 -> 45.1 | No clear effect |
Surprising findings
- Bidirectional modeling performed the same as the autoregressive variant in the ablation (45.1 vs 45.1 at 64/8 frames).
- The 3B model outperforms the 80B Flamingo on MSRVTT-QA despite using far fewer parameters, attributed to the decoupled architecture.
Limitations
- Text (e.g., ASR) concurrent with audio/video is not used and is left to future work.
- Evaluation relies on open-ended exact-match generation, which can penalize correct synonyms.
- Pretraining uses only about 12% of the VTP dataset and ablations use a smaller configuration.
Experimental setup
Datasets: Video-Text Pairs (VTP) MSRVTT-QA ActivityNet-QA NExT-QA Kinetics-Sound VGG-Sound Epic-Sound
Benchmarks: MSRVTT-QA ActivityNet-QA NExT-QA Kinetics-Sound VGG-Sound Epic-Sound
Metrics: accuracy (exact match on open-ended generation)
Baselines: Flamingo GIT2 VideoCoca MBT UAVM MAViL ONE-PEACE UMT-L M-PLUG2 MIST - CLIP
Builds on: Token Turing Machines Flamingo TokenLearner ViT
Closest papers by meaning
- Jointly Training Large Autoregressive Multimodal Models (ICLR 2024 · similarity 0.90)
- FAVE: A Structured Benchmark for Fine-Grained Audio-Visual Temporal Evaluation in Multimodal LLMs (CVPR 2026 · similarity 0.88)
- Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion (ICML 2024 · similarity 0.87)
- Show-o2: Improved Native Unified Multimodal Models (NeurIPS 2025 · similarity 0.87)
- MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis (CVPR 2025 · similarity 0.87)
- Gramian Multimodal Representation Learning and Alignment (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.