Research map / Vision-language and multimodal / Cross-modal learning

X-AVDD: Cross-Attentive Audio-Visual Dataset Distillation

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Audio-visual speech separation

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes X-AVDD, a bi-directional cross-attention audio-visual dataset distillation method that decouples cross-modal alignment via a pre-trained teacher, cutting memory and synthesis cost while improving accuracy over AVDD and a multimodal SRe2L baseline.

Problem

Audio-visual dataset distillation (AVDD) is memory-intensive, failing to scale to higher samples-per-class and complex architectures, and synthetic data often loses cross-modal semantic correspondence (large modality gap).

Approach

Insert bi-directional multi-head cross-attention into a pre-trained teacher, precompute and store real cross-attended embeddings, and distill synthetic data class-wise by matching cross-attended embeddings with a Euclidean distribution-matching loss, so the full synthetic set need not reside in GPU memory.

Key results

• On VGGSound at SPC=10, X-AVDD reaches 24.0% top-1 accuracy versus 8.2% for AVDD (ConvNet); at SPC=20, 26.5% vs 9.8%.
• On AVE with ViT target at SPC=50, X-AVDD achieves 35.3% vs 25.5% for SRe2L-style baseline (ResNet-18 distilled).
• AudioCLIP trained on X-AVDD distilled VGGSound-10K at SPC=50 (5.65% of data) reaches 62.3% vs 72.4% full-data accuracy with 2.74 vs 25.30 hours training time.

Surprising findings

• Class-wise batching of AVDD drops accuracy from 58.0% to 49.5% at SPC=20, showing inter-class information matters for distribution matching.
• At extremely low SPC on ConvNet, the cross-modal constraints can over-regularize and give no benefit over AVDD.

Limitations

• Evaluation limited to audio-visual classification; temporal localization and retrieval left for future work.
• AVDD results for larger architectures are unavailable due to OOM, so baselines for ResNet-18 and ViT are a self-designed multimodal SRe2L adaptation.
• Some results show no gain at very low SPC on ConvNet (e.g., AVE SPC=1 with ConvNet, DM/AVDD comparison mixed).

Experimental setup

Datasets: AVE MUSIC-21 VGGSound VGGSound-10K

Benchmarks: AVE MUSIC-21 VGGSound VGGSound-10K

Metrics: top-1 test accuracy GPU-hours modality gap (class-wise l2 distance) Vendi score intra-class cosine similarity coverage improvement over random (IOR)

Baselines: DM AVDD SRe2L (multimodal adaptation)

Builds on: Distribution Matching AVDD SRe2L

Closest papers by meaning

• V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models (AAAI 2024 · similarity 0.87)
• MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis (CVPR 2025 · similarity 0.87)
• Aligned Better, Listen Better for Audio-Visual Large Language Models (ICLR 2025 · similarity 0.87)
• CrossMAE: Cross-Modality Masked Autoencoders for Region-Aware Audio-Visual Pre-Training (CVPR 2024 · similarity 0.87)
• Asymmetric Hierarchical Anchoring for Robust Audio–Visual Cross-Modal Generalization (NeurIPS 2026 · similarity 0.86)
• Continual Audio-Visual Sound Separation (NeurIPS 2024 · similarity 0.86)