Research map / Vision-language and multimodal / Cross-modal learning
JoVALE: Detecting Human Actions in Video Using Audiovisual and Language Contexts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes JoVALE, a Transformer-based actor-centric network that fuses audio, visual, and BLIP-generated scene-descriptive features for video action detection, reaching state-of-the-art on AVA, UCF101-24, and JHMDB51-21.
Problem
Video action detection must localize and classify actors' actions using multiple information sources (audio, visual, scene context), but how to select and fuse action-relevant multi-modal cues per actor was largely unexplored.
Approach
An Actor-centric Multi-modal Fusion Network (AMFN) initializes action queries from person-detector actor features, jointly encodes each modality's action embeddings with modality context via temporal bottleneck features (MFE), and adaptively fuses per-actor modality embeddings with gated weights (MFA); scene descriptions come from a frozen fine-tuned BLIP captioner's image encoder.
Key results
- On AVA v2.2 with ViT-B backbone (K710 pretrain), JoVALE reaches 40.1 mAP, 2.4 above EVAD (37.7).
- With SF-R101 (K700) on AVA, JoVALE reaches 35.5 mAP, above TubeR's 33.6.
- On UCF101-24 and JHMDB51-21 (visual and scene features only), JoVALE reports 84.9 and 91.0 mAP respectively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use JoVALE instead of TubeR with a 3D-CNN backbone | AVA 2.2 VAD · AVA 2.2 val, 3D-CNN backbone | mAP: 33.6 -> 35.5 | Helped |
| use JoVALE instead of EVAD with a ViT backbone | AVA 2.2 VAD · AVA 2.2 val, ViT backbone | mAP: 37.7 -> 40.1 | Helped |
| use JoVALE instead of the strongest reported baseline, HIT | UCF101-24 VAD · UCF101-24 first split | mAP: 84.8 -> 84.9 | Helped |
| use JoVALE instead of STMixer | JHMDB51-21 VAD · JHMDB51-21, average across three splits | mAP: 86.7 -> 91.0 | Helped |
| add scene-descriptive features to video (ablation) | JoVALE · AVA v2.2, SlowFast-R50 ablation | mAP: 28.0 -> 32.7 | Helped |
| add audio and scene-descriptive features to video (ablation) | JoVALE · AVA v2.2, SlowFast-R50 ablation | mAP: 28.0 -> 34.0 | Helped |
| use audio modality only (ablation) | JoVALE · AVA v2.2, SlowFast-R50 ablation | mAP: 28.0 -> 11.5 | Hurt |
| replace JoVALE fusion with RoI feature fusion (ablation) | JoVALE · AVA v2.2, SlowFast-R50, fusion strategy ablation | mAP: 32.7 -> 29.1 | Hurt |
| replace JoVALE fusion with RoI relation modeling (ablation) | JoVALE · AVA v2.2, SlowFast-R50, fusion strategy ablation | mAP: 32.7 -> 30.4 | Hurt |
| replace JoVALE fusion with global relation modeling (ablation) | JoVALE · AVA v2.2, SlowFast-R50, fusion strategy ablation | mAP: 32.7 -> 32.1 | Hurt |
| replace actor-centric MFE with a factorized encoder (ablation) | JoVALE · AVA v2.2, SlowFast-R50, MFE ablation | mAP: 34.0 -> 30.6 | Hurt |
| replace actor-centric MFE with cross-frame attention (ablation) | JoVALE · AVA v2.2, SlowFast-R50, MFE ablation | mAP: 34.0 -> 32.4 | Hurt |
Surprising findings
- Audio alone yields only 11.5 mAP on AVA, far below video (28.0) or scene-descriptive (26.9) alone.
- Adding audio to video plus scene features improves mAP from 32.7 to 34.0 only modestly, while video plus audio alone gives 28.6.
Limitations
- Higher computational cost (495 GFLOPs vs 243 for EVAD) due to audio and BLIP branches.
- Audio is absent in over 80% of UCF101-24 and JHMDB51-21 clips, so those benchmarks rely mainly on visual and scene features.
- Scene features come from a captioning-finetuned BLIP rather than general-purpose VLMs.
Experimental setup
Datasets: AVA v2.2 UCF101-24 JHMDB51-21 COCO Caption Kinetics-400 Kinetics-700 Kinetics-710
Benchmarks: AVA v2.2 UCF101-24 JHMDB51-21
Metrics: frame-level mAP@0.5
Baselines: EVAD STMixer TubeR MeMViT MVD HIT ACAR AIA WOO SlowFast VideoMAE MViTv2
Builds on: BLIP SlowFast VideoMAE v2 AST adaptive gated fusion
Closest papers by meaning
- Storyboard-guided Alignment for Fine-grained Video Action Recognition (NeurIPS 2025 · similarity 0.86)
- LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos (CVPR 2025 · similarity 0.86)
- Video Action Differencing (ICLR 2025 · similarity 0.86)
- Align Before Adapt: Leveraging Entity-to-Region Alignments for Generalizable Video Action Recognition (CVPR 2024 · similarity 0.86)
- Text-Infused Attention and Foreground-Aware Modeling for Zero-Shot Temporal Action Detection (NeurIPS 2024 · similarity 0.85)
- LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.