Research map / Vision-language and multimodal / Visual question answering
CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarios
Qilang Ye, Zitong Yu, Rui Shao, Xinyu Xie, Philip Torr, Xiaochun Cao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add question-related clue tokens to video and audio tokens (ablation) | CAT · Video-ChatGPT benchmark | Correctness of Information: 40.6 -> 61.6 | Improved |
| add audio tokens to video tokens (ablation) | CAT · Video-ChatGPT benchmark | Correctness of Information: 38.6 -> 40.6 | Improved |
| add visual query h_v in clue aggregator (ablation) | CAT clue aggregator · AVinstruct test set | BLEU-4: 10.0 -> 30.8 | Improved |
| add audio query h_a in clue aggregator (ablation) | CAT clue aggregator · AVinstruct test set | BLEU-4: 18.8 -> 30.8 | Improved |
| add reverse-order block B2 after B1 (ablation) | CAT clue aggregator · AVinstruct test set | BLEU-4: 25.9 -> 30.8 | Improved |
| add ADPO retraining (ablation) | CAT · AVSD | CIDEr: 74.8 -> 79.0 | Improved |
| add ADPO retraining (ablation) | Video-LLaMA · AVSD | CIDEr: 63.7 -> 69.2 | Improved |
| increase number of visual tokens (ablation) | CAT · Video-ChatGPT benchmark | temporal/consistency/detail accuracy | Hurt |
| use 48 learnable queries in Q-former for clue tokens (ablation) | CAT clue aggregator · Video-ChatGPT benchmark | accuracy | Improved |
| use CAT-7B instead of ChatBridge-13B in zero-shot Music-AVQA | audio-visual MLLM · Music-AVQA (zero-shot) | accuracy: 47.6 -> 48.6 | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 2 of 3 on most shared results.
- Chat-UniVi — Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding (CVPR 2024) · better on 0 of 1 shared result
- OneLLM-7B — OneLLM: One Framework to Align All Modalities with Language (CVPR 2024) · better on 1 of 1 shared result
- VAST — VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset (NeurIPS 2023) · better on 1 of 1 shared result
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Question-Aware Gaussian Experts for Audio-Visual Question Answering (CVPR 2025 · similarity 0.90)
- HAVE-Bench: Hierarchical Audio-Visual Evaluation from Perception to Interaction (CVPR 2026 · similarity 0.90)
- OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.90)
- Cross-modal Information Flow in Multimodal Large Language Models (CVPR 2025 · similarity 0.90)
- When Eyes and Ears Disagree: Can MLLMs Discern Audio-Visual Confusion? (AAAI 2026 · similarity 0.90)
- Audio-Visual Adaptive Fusion Network for Question Answering Based on Contrastive Learning (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
