Research map / Video / Video-language understanding
Attention-Based Multimodal Fusion for Video Description
Chiori Hori, Takaaki Hori, Teng-Yok Lee, Ziming Zhang, Bret Harsham, John R. Hershey, Tim K. Marks, Kazuhiko Sumi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add multimodal attention (replacing simple additive fusion) on VGGNet+C3D features | temporal-attention video captioner · YouTube2Text test | CIDEr: 0.665 -> 0.699 | Improved |
| add multimodal attention (replacing simple fusion) on VGGNet+C3D features | temporal-attention video captioner · YouTube2Text test | BLEU4: 0.498 -> 0.507 | Improved |
| add multimodal attention (replacing simple fusion) on VGGNet+C3D features | temporal-attention video captioner · YouTube2Text test | BLEU1: 0.824 -> 0.801 | Hurt |
| add multimodal attention (replacing simple fusion) on VGGNet+C3D+MFCC features | temporal-attention video captioner · YouTube2Text test | BLEU4: 0.510 -> 0.517 | Improved |
| add multimodal attention (replacing simple fusion) on VGGNet+C3D+MFCC features | temporal-attention video captioner · YouTube2Text test | CIDEr: 0.679 -> 0.695 | Improved |
| add MFCC audio features to multimodal attention model (ablation) | Multimodal Attention VGGNet+C3D · YouTube2Text test | CIDEr: 0.699 -> 0.695 | Hurt |
| add MFCC audio features to multimodal attention model (ablation) | Multimodal Attention VGGNet+C3D · YouTube2Text test | BLEU4: 0.507 -> 0.517 | Improved |
| add MFCC audio features to simple multimodal fusion (ablation) | Simple Multimodal VGGNet+C3D · YouTube2Text test | CIDEr: 0.665 -> 0.679 | Improved |
| add C3D motion features to VGGNet with simple multimodal fusion (ablation) | Unimodal VGGNet temporal attention · YouTube2Text test | CIDEr: 0.654 -> 0.665 | Improved |
| use AdaDelta instead of RMSprop for the multimodal attention model (ablation) | Multimodal Attention · YouTube2Text test | CIDEr | Improved |
| use VGGNet instead of GoogLeNet image features in unimodal temporal attention (ablation) | Unimodal temporal attention · YouTube2Text test | CIDEr: 0.568 -> 0.654 | Improved |
Closest papers by meaning
- Mirasol3B: A Multimodal Autoregressive Model for Time-Aligned and Contextual Modalities (CVPR 2024 · similarity 0.86)
- Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders (NeurIPS 2025 · similarity 0.86)
- Adaptive Multimodal Fusion: Dynamic Attention Allocation for Intent Recognition (AAAI 2025 · similarity 0.86)
- Where Does Vision Meet Language? Understanding and Refining Visual Fusion in MLLMs via Contrastive Attention (CVPR 2026 · similarity 0.86)
- Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs (ICML 2026 · similarity 0.86)
- Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
