Research map / Vision-language and multimodal / Cross-modal learning
Aligning Audio-Visual Joint Representations with an Agentic Workflow
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AVAgent, an LLM-driven agentic workflow (tool use, planning, reflection) that edits audio to better align with video before audio-visual MAE pretraining, reporting gains across classification, localization, segmentation and separation.
Problem
Audio-visual pairs in real data often have background noise and temporal non-synchronization, which degrades the quality of learned joint audio-visual representations.
Approach
A data-centric agentic workflow in which a multimodal LLM captions audio and video separately, a fine-tuned Vicuna plans predefined noise-filtering and audio-coordination edits, and ImageBind-based scores reflect on the result in cycles to progressively align audio to visual content before representation learning.
Key results
- On VGGSound-All, AVAgent reaches 61.56% linear and 69.24% finetune accuracy, versus 60.82%/67.61% for AV-MAE and 57.36%/67.17% for MAViL.
- On AVSBench, mIoU 36.37 and F1 49.85, outperforming Mix-and-Localize (31.69 mIoU, 45.35 F1).
- Random audio edits instead of LLM-planned actions drop VGGSound-All linear accuracy from 61.56% to 50.26%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AVAgent-aligned data instead of MAViL's data | Audio-visual representation learning · VGGSound-Music | Linear accuracy (%): 61.95 -> 64.57 | Helped |
| Use AVAgent-aligned data instead of MAViL's data | Audio-visual representation learning · VGGSound-All | Finetune accuracy (%): 67.17 -> 69.24 | Helped |
| Use AVAgent-aligned data instead of Mix-and-Localize | Sound source localization · Flickr-SoundNet | Precision: 55.83 -> 58.23 | Helped |
| Use AVAgent-aligned data instead of Mix-and-Localize | Sound source segmentation · AVSBench | mIoU: 31.69 -> 36.37 | Helped |
| Use AVAgent-aligned data instead of OneAVM | Sound source separation · VGGSound-Music | SDR: 2.51 -> 5.26 | Helped |
| Remove LoRA tuning (equivalently, add LoRA tuning) (ablation) | AVAgent · VGGSound-All | Linear accuracy (%): 59.87 -> 61.56 | Helped |
| Remove LoRA tuning (equivalently, add LoRA tuning) (ablation) | AVAgent · Flickr-SoundNet | Precision: 56.93 -> 58.23 | Helped |
| Remove LoRA tuning (equivalently, add LoRA tuning) (ablation) | AVAgent · AVSBench | mIoU: 33.79 -> 36.37 | Helped |
| Replace LLM-guided audio edits with random audio modifications | AVAgent · VGGSound-All | Linear accuracy (%): 50.26 -> 61.56 | Helped |
| Replace LLM-guided audio edits with random audio modifications | AVAgent · AVSBench | mIoU: 22.18 -> 36.37 | Helped |
Limitations
- Effectiveness depends on input quality; very noisy or low-quality inputs can compromise the LLM and VLM.
- Substantial computational cost of LLM and VLM components limits real-time use.
- Generalization to extremely noisy or acoustically complex environments not extensively verified.
Experimental setup
Datasets: VGGSound VGGSound-Music VGGSound-All VGGSound-Instruments MUSIC Flickr-SoundNet AVSBench AudioSet
Benchmarks: VGGSound-Music VGGSound-All AudioSet Flickr-SoundNet AVSBench MUSIC VGGS-Instruments
Metrics: top-1 accuracy Precision AP F1 mIoU SDR SAR alignment score T-Alignment
Baselines: MAE AudioMAE CAV-MAE MAViL AV-MAE Attention 10k Mix-and-Localize SLAVC OneAVM CCoL
Builds on: masked autoencoder CAV-MAE ImageBind LoRA
Closest papers by meaning
- Aligned Better, Listen Better for Audio-Visual Large Language Models (ICLR 2025 · similarity 0.90)
- Native Audio-Visual Alignment for Generation (NeurIPS 2026 · similarity 0.88)
- MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis (CVPR 2025 · similarity 0.87)
- CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment (CVPR 2025 · similarity 0.87)
- Asymmetric Hierarchical Anchoring for Robust Audio–Visual Cross-Modal Generalization (NeurIPS 2026 · similarity 0.86)
- Foley-Flow: Coordinated Video-to-Audio Generation with Masked Audio-Visual Alignment and Dynamic Conditional Flows (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.