AI Paper Atlas

Research map / Video / Video-language understanding

ECLIPSE: Efficient Long-Range Video Retrieval Using Sight and Sound

Yan-Bo Lin, Jie Lei, Mohit Bansal, Gedas Bertasius

ECCV 2022 · Video › Video-language understanding · approach: Video LLMs with fine-grained understanding · 46 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add audio via audiovisual attention to CLIP4ClipCLIP4Clip · ActivityNet Captions (val1)T2V R@1: 40.7 -> 42.3Improved
replace 96 video frames with 32 frames plus audioCLIP4Clip · ActivityNet Captions (val1)efficiency: 1251 -> 827 GFLOPs, 17.39 -> 50.93 samples/sec, 41.7 -> 42.3 R@1Improved
add audio on top of video (vs 64-frame video-only) (ablation)ECLIPSE · ActivityNet Captions (val1)T2V R@1: 40.7 -> 42.3 (audio-only 2.7)Improved
use ViT-B/16 instead of ViT-B/32 (ablation)ECLIPSE · ActivityNet Captions (val1)T2V R@1: 42.3 -> 45.3Improved
use dual-pathway A2V+V2A attention instead of A2V only or joint AV (ablation)ECLIPSE · ActivityNet Captions (val1) (8 frames)T2V R@1Improved
use 30 s audio instead of 10 s (ablation)ECLIPSE · ActivityNet Captions (val1)T2V R@1: +0.5% R@1Improved
change the audio encoder (VGGish, VGGSound, AST) (ablation)ECLIPSE · ActivityNet Captions (val1)T2V R@1No clear effect
insert audiovisual block in all 12 layers instead of fewer (ablation)ECLIPSE · ActivityNet Captions (val1)T2V R@1Improved
CLIP pretraining instead of ImageNet-21k pretraining (ablation)ECLIPSE · ActivityNet, DiDeMo, YouCook2, QVHighlightsT2V R@1: 27.1%, 34.6%, 24.2%, 42.5% betterImproved
random instead of uniform frame sampling at 32 frames (ablation)CLIP4Clip (video-only) · ActivityNet Captions (val1)T2V R@1: 40.4 -> 41.0Improved
random instead of uniform frame sampling at 96 frames (ablation)CLIP4Clip (video-only) · ActivityNet Captions (val1)T2V R@1: 41.7 -> 40.9Hurt
remove overlapping VGGSound test videos from ActivityNet (ablation)ECLIPSE and CLIP4Clip · ActivityNet Captions (val1)T2V R@1: 42.3 -> 42.3 and 40.7 -> 40.8No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy