Research map / Video / Action recognition
Learnable Irrelevant Modality Dropout for Multimodal Action Recognition on Modality-Specific Annotated Videos
Saghir Alfasly, Jian Lu, Chen Xu, Yuru Zou
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use learnable relevance-based audio masking and multimodal fusion | IMD action recognition · Kinetics400 | Top-1: 80.9 -> 82.3 | Improved |
| Use learnable relevance-based audio masking and multimodal fusion | IMD-B · UCF-101 | Top-1: 98.9 -> 99.1 | Improved |
| Use learnable relevance-based audio masking and multimodal fusion | IMD-B · HMDB51 | Top-1: 70.2 -> 71.3 | Improved |
| Use learnable relevance-based audio masking and multimodal fusion | IMD-B · Kinetics-Sounds | Top-1: 85.0 -> 90.48 | Improved |
| Use BERT rather than word2vec or GloVe for SAVLD label embeddings (ablation) | SAVLD · Kinetics400 | Top-1: 80.9 -> 82.8; 81.2 -> 82.8 | Improved |
| Use cosine rather than Euclidean similarity for BERT label matching (ablation) | SAVLD · Kinetics400 | Top-1: 82.8 -> 82.8 | No clear effect |
| Add audio and flow to RGB (ablation) | Proposed multimodal framework · Kinetics400 | Top-1: 79.1 -> 82.8 | Improved |
| Use IOU rather than Dice overlap loss (ablation) | IMD · Kinetics400 | Top-1: 81.6 -> 81.7 | No clear effect |
| Set the K-Y_a ratio to 10-20 (ablation) | IMD relevance ground truth · Kinetics400 | Top-1: 81.7 -> 82.3 | Improved |
| Set the relevance threshold alpha to 0.25 rather than 0.75 (ablation) | IMD · Kinetics400 | Top-1: 80.9 -> 82.8 | Improved |
| Mask irrelevant audio instead of weighting audio at every relevance score (ablation) | IMD · Kinetics400 | Top-1: 80.4 -> 82.8 | Improved |
| Add intra-class cross-modality augmentation (ablation) | IMD · Kinetics400 | Top-1: 82.3 -> 82.8 | Improved |
Closest papers by meaning
- Modality-Independent Teachers Meet Weakly-Supervised Audio-Visual Event Parser (NeurIPS 2023 · similarity 0.89)
- Towards Good Practices for Missing Modality Robust Action Recognition (AAAI 2023 · similarity 0.87)
- A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition (CVPR 2024 · similarity 0.86)
- A Multimodal, Multi-Task Adapting Framework for Video Action Recognition (AAAI 2024 · similarity 0.86)
- Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities (CVPR 2024 · similarity 0.85)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
