Research map / Vision-language and multimodal / Cross-modal learning

Asymmetric Hierarchical Anchoring for Robust Audio–Visual Cross-Modal Generalization

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Self-supervised audio-visual representation learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Asymmetric Hierarchical Anchoring (AHA), which uses audio RVQ's primary layer as a shared semantic anchor for video distillation, plus a GRL adversarial decoupler and Local Sliding Alignment, improving zero-shot cross-modal generalization on AVE/AVVP/UCF-VGG.

Problem

Cross-modal generalization (CMG) in audio-visual joint discrete representation learning suffers from information allocation ambiguity in symmetric frameworks, where semantic information leaks into modality-specific branches and MI estimators are unreliable.

Approach

Designate the first RVQ layer(s) of the audio branch as a directional semantic anchor that video semantic features must align to, replace CLUB-style MI minimization with a GRL-based adversarial contrastive decoupler, and add a local sliding alignment loss for fine-grained temporal matching.

Key results

• AHA averages 62.24 across eight CMG transfer settings versus 56.11 for Unicode (+6.13).
• On AVVP V→A, AHA reaches 73.4 F1 vs 59.7 for Unicode (+13.7).
• On the talking-face experiment, AHA achieves PSNR 29.42 and LPIPS 0.0468, versus 26.46 PSNR for the variant without GRL loss.

Surprising findings

• AHA underperforms the DCID/FCID AVT baselines on AVE→AVVP (52.5/51.3 vs 57.4/58.5 for FCID), and the paper acknowledges this.
• Replacing the GRL decoupler with a CLUB estimator yields a clear drop, supporting the claim that variational MI upper bounds are less effective here.

Limitations

• Fixed semantic anchor; adaptive anchor selection not explored.
• Only audio-visual CMG and a diagnostic talking-face setting evaluated.
• Underperforms AVT baselines on AVE to AVVP transfer.

Experimental setup

Datasets: VGGSound-AVEL 40K AVE AVVP UCF-101 TalkVid

Benchmarks: AVE AVVP AVE to AVVP UCF(v) VGG(a)

Metrics: accuracy F1 V2V-LS Mouth RMSE PSNR LPIPS

Baselines: MST CODIS TURN CMCM Unicode DCID FCID

Builds on: Unicode CLUB Cross-CPC LIA DCID

Closest papers by meaning

• From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.87)
• CrossMAE: Cross-Modality Masked Autoencoders for Region-Aware Audio-Visual Pre-Training (CVPR 2024 · similarity 0.87)
• ES3: Evolving Self-Supervised Learning of Robust Audio-Visual Speech Representations (CVPR 2024 · similarity 0.87)
• Native Audio-Visual Alignment for Generation (NeurIPS 2026 · similarity 0.87)
• CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment (CVPR 2025 · similarity 0.87)
• Seeing Speech and Sound: Distinguishing and Locating Audio Sources in Visual Scenes (CVPR 2025 · similarity 0.87)