Research map / Video / Video-language understanding
Task-Aware Mechanism: Hybrid MoE Vision Tower Towards Holistic Video Understanding
Qishen Yin, Tanghui Jia, Peng Jin, Li Hao, Juntong Wu, Guanlin Lu, Li Yuan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add MoE vision encoder alone (no DFR) (ablation) | Qwen2-7B LVLM baseline · ActivityNet-QA, MVBench, MLVU-Test | avg relative change: -2.7% | Hurt |
| add Dynamic Frame Number & Resolution alone (ablation) | Qwen2-7B LVLM baseline · ActivityNet-QA, MVBench, MLVU-Test | avg relative change: +2.8% | Improved |
| combine MoE vision encoder with DFR (ablation) | TallVA · ActivityNet-QA, MVBench, MLVU-Test | avg relative change: +12.1% | Improved |
| add DFR and Time/Task instruction to Inductor routing (ablation) | baseline uniform budget · MLVU-Test | accuracy: 44.8 -> 46.1 | Improved |
| train the LLM first in Stage 2 (ablation) | TallVA · MVBench | accuracy: 59.5 -> 61.8 | Improved |
| train Projector only or VE and Projector in Stage 2 (ablation) | TallVA · MLVU-Test | accuracy: 46.1 -> 42.2 (Projector); 46.1 -> 32.7 (VE and Projector) | Hurt |
| soft-gated MoE vision encoder (top2 in 4) (ablation) | TallVA after stage 2 · MVBench | accuracy: 61.8 -> 63.9 | Improved |
| hard-gate vision encoder by task type (ablation) | TallVA after stage 2 · MVBench | accuracy: 61.8 -> 56.4 | Hurt |
| increase soft MoE-VE experts from 4 to 8 (top2) (ablation) | TallVA · MVBench | accuracy: 63.9 -> 63.2 | Hurt |
| hard resolution-based MoE projector (ablation) | TallVA after stage 2 · MVBench | accuracy: 61.8 -> 62.6 | Improved |
| soft-gated MoE projector (ablation) | TallVA after stage 2 · MVBench | accuracy: 61.8 -> 61.9 (top2 in 4); 61.8 -> 61.6 (top2 in 8) | No clear effect |
| sparse MoE vision encoder without DFR (Top2 in 8) (ablation) | dense baseline · MVBench | accuracy: 58.6 -> 53.3 | Hurt |
Closest papers by meaning
- MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understanding (ECCV 2026 · similarity 0.89)
- Think-as-You-See: Streaming Chain-of-Thought Reasoning for Large Vision-Language Models (CVPR 2026 · similarity 0.86)
- BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding (CVPR 2025 · similarity 0.86)
- Leveraging Textual Memory and Key Frame Reasoning for Full Video Understanding Using Off-the-Shelf LLMs and VLMs (Student Abstract) (AAAI 2025 · similarity 0.85)
- HierarQ: Task-Aware Hierarchical Q-Former for Enhanced Video Understanding (CVPR 2025 · similarity 0.85)
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
