Research map / Vision-language and multimodal / Cross-modal learning
Mixtures of Experts for Audio-Visual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AVMoE, which injects unimodal and cross-modal adapters as experts into frozen audio and visual transformers, combined by a lightweight router, for parameter-efficient audio-visual learning across AVE, AVVP, AVS and AVQA.
Problem
Parameter-efficient transfer of pre-trained audio and visual models to audio-visual tasks is limited by cross-modal adapters alone, which can inject irrelevant information when modalities are misaligned or missing.
Approach
Combine a cross-modal adapter (LAVisH-style latent-token cross-attention fusion) and a unimodal self-attention adapter as MoE experts in frozen backbones, with a token-level MLP router (softmax weights plus Gaussian noise during training) that dynamically weights them per input.
Key results
- On AVE with Swin-V2-L and HTS-AT audio, AVMoE reaches 82.6% accuracy versus 82.2% for DG-SCT and 78.6% for LAVisH.
- On MUSIC-AVQA, AVMoE averages 75.7% versus 74.8% for DG-SCT, with 71.9% on audio-visual questions.
- On AVSBench MS3, AVMoE gets MJ 54.5 and MF 68.7 versus 53.5 and 64.2 for DG-SCT; visual-only AVE accuracy is 78.4% versus 72.1% for DG-SCT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace DG-SCT with AVMoE | Swin-V2-L + HTS-AT on AVE · AVE test | accuracy: 82.2 -> 82.6 | Helped |
| Replace DG-SCT with AVMoE | AVMoE on AVSBench MS3 · AVSBench MS3 | MF: 64.2 -> 68.7 | Helped |
| Replace DG-SCT with AVMoE | Swin-V2-L + HTS-AT on MUSIC-AVQA · MUSIC-AVQA test | Avg accuracy: 74.8 -> 75.7 | Helped |
| Add a second unimodal adapter while keeping one cross-modal adapter (ablation) | AVMoE on AVSBench MS3 · AVSBench MS3 | MF: 67.5 -> 68.1 | Helped |
| Add a second cross-modal adapter while keeping one unimodal adapter (ablation) | AVMoE on AVSBench MS3 · AVSBench MS3 | MF: 67.5 -> 68.4 | Helped |
| Add a second cross-modal adapter while keeping one unimodal adapter (ablation) | AVMoE on AVE · AVE | accuracy: 81.3 -> 82.3 | Helped |
| Add a second unimodal adapter while keeping one cross-modal adapter (ablation) | AVMoE on AVE · AVE | accuracy: 81.3 -> 81.8 | Helped |
| Test with visual-only rather than audio-visual input (ablation) | AVMoE versus DG-SCT on AVE · AVE visual-only | accuracy: 72.1 -> 78.4 | Helped |
| Test with visual-only rather than audio-visual input (ablation) | AVMoE versus DG-SCT on AVS · AVS MF visual-only | MF: 79.1 -> 85.4 | Helped |
| Test with visual-only rather than audio-visual input (ablation) | AVMoE versus DG-SCT on AVQA · AVQA visual-only | accuracy: 67.3 -> 73.2 | Helped |
Surprising findings
- Visual-only inference on AVE drops only from 82.6% to 78.4% for AVMoE, while DG-SCT drops from 82.2% to 72.1%.
Limitations
- Only a small number of experts (one or two per type) were explored due to GPU memory limits.
- No statistical significance or error bars were reported because single runs are expensive.
- Trainable parameter fraction is large (around 30-40%), which weakens the parameter-efficiency claim.
Experimental setup
Datasets: AVE LLP AVSBench MUSIC-AVQA
Benchmarks: AVE LLP AVSBench MUSIC-AVQA
Metrics: accuracy F-score Jaccard index J F-score F
Baselines: LAVisH DG-SCT AVEL AVSDN MGN AVS ST-AVQA Pano-AVQA AVSD
Builds on: LAVisH DG-SCT Mixture of Experts Swin-Transformer HTS-AT
Closest papers by meaning
- Few-Shot Audio-Visual Class-Incremental Learning with Temporal Prompting and Regularization (AAAI 2025 · similarity 0.88)
- Continual Audio-Visual Sound Separation (NeurIPS 2024 · similarity 0.87)
- TSAM: Temporal SAM Augmented with Multimodal Prompts for Referring Audio-Visual Segmentation (CVPR 2025 · similarity 0.87)
- Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation (CVPR 2024 · similarity 0.87)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.87)
- Adapting to the Unknown: Training-Free Audio-Visual Event Perception with Dynamic Thresholds (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.