Research map / Vision-language and multimodal / Cross-modal learning
Crab: A Unified Audio-Visual Scene Understanding Model with Explicit Cooperation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Crab unifies temporal, spatial, spatio-temporal and pixel-level audio-visual understanding in one LLaMA-2-based model, trained on a refined instruction dataset with explicit reasoning (AV-UIE) and an interaction-aware multi-head LoRA.
Problem
Existing audio-visual models handle one task family at a time, and naive joint multi-task instruction tuning causes interference across heterogeneous audio-visual tasks because labels are simple words with no explicit inter-task relationship.
Approach
Augment existing AV datasets with MLLM-generated explicit reasoning chains (AV-UIE) so tasks share intermediate temporal/spatial information, and replace standard LoRA with an interaction-aware LoRA (shared A matrix, multiple B heads with a router) intended to specialize in temporal, spatial and pixel-level interaction aspects.
Key results
- Crab reaches 78.94% average accuracy on MUSIC-AVQA, beating TSPM (76.79%) and other specialized models.
- On AVE, Crab reports 80.15% accuracy versus 77.80% for MM-Pyramid.
- Ablation: removing explicit reasoning process drops MUSIC-AVQA accuracy from 78.94 to 76.05, and replacing IA-LoRA with standard LoRA drops it to 76.92.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| include explicit reasoning process instead of simple labels (ablation) | Crab · AVQA | Acc: 76.05 -> 78.94 | Helped |
| include explicit reasoning process instead of simple labels (ablation) | Crab · AVE | Acc: 78.62 -> 80.15 | Helped |
| include explicit reasoning process instead of simple labels (ablation) | Crab · AVVP | Segment-level F1: 52.01 -> 59.00 | Helped |
| include explicit reasoning process instead of simple labels (ablation) | Crab · AVVP | Event-level F1: 51.36 -> 54.44 | Helped |
| replace standard LoRA with interaction-aware LoRA (ablation) | Crab · AVQA | Acc: 76.92 -> 78.94 | Helped |
| replace standard LoRA with interaction-aware LoRA (ablation) | Crab · AVE | Acc: 79.93 -> 80.15 | Helped |
| replace standard LoRA with interaction-aware LoRA (ablation) | Crab · AVVP | Segment-level F1: 53.54 -> 59.00 | Helped |
| replace standard LoRA with interaction-aware LoRA (ablation) | Crab · AVVP | Event-level F1: 53.15 -> 54.44 | Helped |
| remove LoRA head-1 (ablation) | Crab interaction-aware LoRA · AVE | Acc: 80.15 -> 75.07 | Hurt |
| remove LoRA head-2 (ablation) | Crab interaction-aware LoRA · ARIG | cIoU: 41.78 -> 29.16 | Hurt |
| remove LoRA head-3 (ablation) | Crab interaction-aware LoRA · ARIG | cIoU: 41.78 -> 38.65 | Hurt |
Surprising findings
- Each LoRA head shows a distinct task dependence, with heads loosely mapping to temporal, spatial and pixel-level abilities.
- Explicit reasoning helps audio-visual QA and event tasks much more than pixel-level segmentation tasks.
Limitations
- Accuracy of generated explicit reasoning processes is not guaranteed and can hurt performance.
- Pixel-level understanding remains a weaker area; lags PVT-v2-based specialized models on several AVS subtasks.
- Uses ViT-L/14 instead of segmentation-specific backbones, limiting segmentation performance.
Experimental setup
Datasets: MUSIC-AVQA AVE LLP AVS-Bench Ref-AVS VALOR Video-LLaVA pre-training data AudioCaps LVIS
Benchmarks: AVE LLP/AVVP AVS-Bench S4 AVS-Bench MS3 AVS-Bench AVSS Ref-AVS MUSIC-AVQA
Metrics: accuracy segment-level F1 event-level F1 cIoU AUC mIoU F-score
Baselines: TimeChat MEERKAT GroundingGPT X-InstructBLIP VALOR AnyRef AVSBench AVSegFormer TSPM MM-Pyramid
Builds on: LoRA HydraLoRA SAM BEATs Video-LLaVA
Closest papers by meaning
- Hear, Localize, and Reason: Spatially Aware Scene Understanding for Audio-visual LLMs (NeurIPS 2026 · similarity 0.90)
- The World is Not Mono: Enabling Spatial Understanding in Large Audio-Language Models (NeurIPS 2026 · similarity 0.89)
- Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer (AAAI 2024 · similarity 0.89)
- SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing (NeurIPS 2025 · similarity 0.89)
- JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation (ICLR 2026 · similarity 0.88)
- Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.