Research map / Vision-language and multimodal / Cross-modal learning
Audio-Visual Generalised Zero-Shot Learning With Cross-Modal Attention and Language
Otniel-Bogdan Mercea, Lukas Riesch, A. Sophia Koepke, Zeynep Akata
Check your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train jointly with audio and visual inputs rather than a single modality (ablation) | AVCA · ActivityNet-GZSL | HM: 7.53 -> 12.13 | Improved |
| Train jointly with audio and visual inputs rather than a single modality (ablation) | AVCA · UCF-GZSL | HM: 20.92 -> 27.15 | Improved |
| Remove cross-attention (ablation) | AVCA · ActivityNet-GZSL | HM: 12.13 -> 6.50 | Hurt |
| Remove audio-branch training components while retaining cross-attention (ablation) | AVCA · VGGSound-GZSL | HM: 6.31 -> 6.63 | Mixed |
| Use audio-branch output rather than visual-branch output for evaluation (ablation) | AVCA · ActivityNet-GZSL | HM: 12.13 -> 12.54 | Mixed |
| Combine audio- and visual-branch outputs at evaluation rather than use the visual output (ablation) | AVCA · UCF-GZSL | HM: 27.15 -> 25.78 | Hurt |
| Remove base triplet loss l_t (ablation) | AVCA · ActivityNet-GZSL | HM: 12.13 -> 8.39 | Hurt |
| Remove base triplet loss l_t (ablation) | AVCA · ActivityNet-GZSL | ZSL: 9.13 -> 9.54 | Improved |
| Remove reconstruction loss l_rec (ablation) | AVCA · ActivityNet-GZSL | HM: 12.13 -> 9.59 | Hurt |
| Remove decoded triplet loss l_ct (ablation) | AVCA · UCF-GZSL | HM: 27.15 -> 17.88 | Hurt |
| Remove word-alignment triplet loss l_w (ablation) | AVCA · UCF-GZSL | HM: 27.15 -> 20.75 | Hurt |
| Remove regularisation loss l_r (ablation) | AVCA · UCF-GZSL | HM: 27.15 -> 21.31 | Hurt |
Closest papers by meaning
- Temporal and Cross-Modal Attention for Audio-Visual Zero-Shot Learning (ECCV 2022 · similarity 0.95)
- An eye for an ear: zero-shot audio description leveraging an image captioner with audio-visual token distribution matching (NeurIPS 2024 · similarity 0.89)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.89)
- Few-Shot Audio-Visual Class-Incremental Learning with Temporal Prompting and Regularization (AAAI 2025 · similarity 0.88)
- Few-Shot Incremental Learning via Foreground Aggregation and Knowledge Transfer for Audio-Visual Semantic Segmentation (AAAI 2025 · similarity 0.87)
- CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
