AI Paper Atlas

Research map / Video / Video-language understanding

Attention-Based Multimodal Fusion for Video Description

Chiori Hori, Takaaki Hori, Teng-Yok Lee, Ziming Zhang, Bret Harsham, John R. Hershey, Tim K. Marks, Kazuhiko Sumi

ICCV 2017 · Video › Video-language understanding · approach: Encoder-decoder video captioning · 407 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add multimodal attention (replacing simple additive fusion) on VGGNet+C3D featurestemporal-attention video captioner · YouTube2Text testCIDEr: 0.665 -> 0.699Improved
add multimodal attention (replacing simple fusion) on VGGNet+C3D featurestemporal-attention video captioner · YouTube2Text testBLEU4: 0.498 -> 0.507Improved
add multimodal attention (replacing simple fusion) on VGGNet+C3D featurestemporal-attention video captioner · YouTube2Text testBLEU1: 0.824 -> 0.801Hurt
add multimodal attention (replacing simple fusion) on VGGNet+C3D+MFCC featurestemporal-attention video captioner · YouTube2Text testBLEU4: 0.510 -> 0.517Improved
add multimodal attention (replacing simple fusion) on VGGNet+C3D+MFCC featurestemporal-attention video captioner · YouTube2Text testCIDEr: 0.679 -> 0.695Improved
add MFCC audio features to multimodal attention model (ablation)Multimodal Attention VGGNet+C3D · YouTube2Text testCIDEr: 0.699 -> 0.695Hurt
add MFCC audio features to multimodal attention model (ablation)Multimodal Attention VGGNet+C3D · YouTube2Text testBLEU4: 0.507 -> 0.517Improved
add MFCC audio features to simple multimodal fusion (ablation)Simple Multimodal VGGNet+C3D · YouTube2Text testCIDEr: 0.665 -> 0.679Improved
add C3D motion features to VGGNet with simple multimodal fusion (ablation)Unimodal VGGNet temporal attention · YouTube2Text testCIDEr: 0.654 -> 0.665Improved
use AdaDelta instead of RMSprop for the multimodal attention model (ablation)Multimodal Attention · YouTube2Text testCIDErImproved
use VGGNet instead of GoogLeNet image features in unimodal temporal attention (ablation)Unimodal temporal attention · YouTube2Text testCIDEr: 0.568 -> 0.654Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy