Research map / Video / Video generation
MUGEN: A Playground for Video-Audio-Text Multimodal Understanding and GENeration
Thomas Hayes, Songyang Zhang, Xi Yin, Guan Pang, Sasha Sheng, Harry Yang, Songwei Ge, Qiyuan Hu, Devi Parikh
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| query with video-retrieval A+T ensemble instead of audio alone | video retrieval · MUGEN video retrieval | Recall@1: 58.59 -> 81.50 | Improved |
| use ensemble of separate models instead of joint model (A+T*) | video retrieval · MUGEN video retrieval | Recall@1: 62.54 -> 81.50 | Improved |
| use ensemble of separate models instead of joint model (V+T*) | audio retrieval · MUGEN audio retrieval | Recall@1: 41.83 -> 69.59 | Improved |
| add video query to text query for audio retrieval | audio retrieval · MUGEN audio retrieval | Recall@1: 2.40 -> 69.59 | Improved |
| raise video frame rate from 8 to 32 frames | T2V generation · MUGEN T2V (full train) | FVD: 112.7 -> 61.0 | Improved |
| raise video frame rate from 8 to 32 frames | V2T generation · MUGEN V2T | R.Sim.: 83.5 -> 108.0 | Improved |
| use auto-text instead of manual text | T2V generation · MUGEN T2V (full train) | FVD: 61.0 -> 140.7 | Hurt |
| add auto-text to manual text | T2V generation · MUGEN T2V (full train) | FVD: 61.0 -> 61.4 | No clear effect |
| train on down-sampled balanced set instead of full set | T2V generation · MUGEN T2V | R.Sim.: 64.9 -> 70.6 | Improved |
| lower audio compression (more tokens, 68 to 551) | A2T generation · MUGEN A2T | CIDEr: 14.1 -> 15.5 | Improved |
| lower audio compression (more tokens) 68 to 551 | V2A generation · MUGEN V2A | FAD: 523.8 -> 50.0 | Improved |
Closest papers by meaning
- MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos (NeurIPS 2026 · similarity 0.85)
- InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation (ICLR 2024 · similarity 0.84)
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.84)
- GlitchBench: Can Large Multimodal Models Detect Video Game Glitches? (CVPR 2024 · similarity 0.84)
- Mellow: a small audio language model for reasoning (NeurIPS 2025 · similarity 0.84)
- SeriesBench: A Benchmark for Narrative-Driven Drama Series Understanding (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
