Research map / Vision-language and multimodal / Cross-modal learning
AGAV-Rater: Adapting Large Multimodal Model for AI-Generated Audio-Visual Quality Assessment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds AGAVQA-3k, a dataset of 3,382 AI-generated audio-visual clips from 16 video-to-audio methods with multi-dimensional MOS and optimal-pair labels, and trains AGAV-Rater, a VideoLLaMA2-based LMM pretrained on text-defined levels then fine-tuned to regress numerical quality scores.
Problem
Automatically assessing the quality of AI-generated audio-visual content (AGAV) produced by video-to-audio methods, including unique AIGC distortions such as unnatural audio and audio-visual inconsistency, which existing AVQA and AQA methods and audio-visual LMMs handle poorly.
Approach
Adapt a pretrained audio-visual LMM (VideoLLaMA2) into a quality scorer by first pretraining with automatically labeled excellent/bad responses built from real audio-caption datasets (reversed audio and mismatched pairs as negatives), then fine-tuning with a multi-dimensional instruction and a PLCC loss that regresses the LLM's last hidden states to numerical scores.
Key results
- On AGAVQA-MOS, AGAV-Rater reaches SRCC 0.7909 for audio quality, 0.7553 for content consistency, and 0.7458 for overall quality, exceeding the best baseline VAST (0.7640, 0.6811, 0.7094).
- On AGAVQA-Pair optimal-AGAV selection, AGAV-Rater attains 0.78 overall accuracy, versus 0.64 for VAST and 0.40 for the best audio-visual LMM (VideoLLaMA2 single-input).
- In a subjective test on ElevenLabs outputs, 80% of 10 subjects preferred the AGAV selected by AGAV-Rater.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train AGAV-Rater with text-defined-level pretraining (ablation) | AGAV-Rater · AGAVQA-MOS; TTA; TTM | SRCC: 0.7141 -> 0.7458 (AGAVQA-MOS overall); 0.7040 -> 0.7390 (TTA audio quality); 0.7042 -> 0.7719 (TTM content consistency) | Helped |
| fine-tune on numerical scores instead of text-defined levels (ablation) | AGAV-Rater · AGAVQA-MOS; TTA; TTM | SRCC: 0.7078 -> 0.7458 (AGAVQA-MOS overall); 0.7005 -> 0.7390 (TTA audio quality); 0.7359 -> 0.7719 (TTM content consistency) | Helped |
| use multi-dimension rather than single-dimension instructions (ablation) | AGAV-Rater · AGAVQA-MOS; TTA; TTM | SRCC: 0.6865 -> 0.7458 (AGAVQA-MOS overall); 0.7143 -> 0.7390 (TTA audio quality); 0.7565 -> 0.7719 (TTM content consistency) | Mixed |
| use VideoLLaMA2 rather than OneLLM or GroundingGPT as the base model (ablation) | AGAV-Rater · AGAVQA-MOS | SRCC (audio quality): 0.6578 -> 0.7909; 0.4387 -> 0.7909 | Helped |
| fine-tune AGAV-Rater on AGAVQA-MOS | AGAV-Rater · AGAVQA-Pair (All) | accuracy: 0.40 -> 0.78 | Helped |
Surprising findings
- Audio-visual LMMs without fine-tuning perform poorly (SRCC near zero for several models), and closed-source LMMs such as Gemini-1.5 Pro achieve only 0.23 accuracy on optimal-AGAV selection, near chance levels for some categories.
- Replacing numerical MOS regression targets with five text-defined levels slightly reduces performance.
Limitations
- AGAVQA-Pair contains only 75 question-answer pairs, a small evaluation set.
- Audio-visual LMM baselines are evaluated by converting level-token probabilities to scores, which may disadvantage them.
- Subjective study covers only 15 raters and 8 VTA methods in the MOS subset.
Experimental setup
Datasets: VGGSound AudioCaps MusicCaps AGAVQA-3k TTA dataset TTM dataset FETV VBench
Benchmarks: AGAVQA-MOS AGAVQA-Pair TTA dataset TTM dataset
Metrics: SRCC KRCC PLCC RMSE answer accuracy
Baselines: PandaGPT NextGPT VITA-1.0 VideoLLaMA2 MOSNet STOI-Net NISQA PAM AVID-CMA VAST VALOR CLAP
Builds on: VideoLLaMA2 VGGSound AudioCaps MusicCaps
Closest papers by meaning
- AIGV-Assessor: Benchmarking and Evaluating the Perceptual Quality of Text-to-Video Generation with LMM (CVPR 2025 · similarity 0.89)
- AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generation (ICML 2026 · similarity 0.85)
- $\texttt{AVROBUSTBENCH}$: Benchmarking the Robustness of Audio-Visual Recognition Models at Test-Time (NeurIPS 2025 · similarity 0.85)
- Adaptive Image Quality Assessment via Teaching Large Multimodal Model to Compare (NeurIPS 2024 · similarity 0.85)
- TTA-Bench: A Comprehensive Benchmark for Evaluating Text-to-Audio Models (AAAI 2026 · similarity 0.84)
- Massive Sound Embedding Benchmark (MSEB) (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.