Research map / Detection and segmentation / Tracking
Patch-level Sounding Object Tracking for Audio-Visual Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PSOT, a graph-based patch-level tracking model for audio-visual question answering that combines motion-driven, sound-driven, and question-driven key patch selection, reaching 78.42% average accuracy on MUSIC-AVQA.
Problem
Audio-visual question answering requires identifying and tracking the sounding objects relevant to a question along the timeline, which is hard when audio is unclear, objects share appearance, and sounding objects change over time.
Approach
Three parallel-then-sequential graph modules operate on CLIP patch features: a motion-driven KPT uses patch-wise feature-change intensity to build adjacency, a sound-driven KPT uses audio-visual correspondence to build adjacency, and a question-driven KPT keeps the top-r% question-relevant patches before a multimodal message aggregation module predicts the answer.
Key results
- PSOT with CLIP-ViT-L/14 and CLAP reaches 78.42% average accuracy on MUSIC-AVQA, competitive with VALOR (78.90%) which uses 33.5M pretraining pairs.
- Removing M-KPT, S-KPT, or Q-KPT lowers average accuracy to 77.79%, 77.57%, or 77.65% respectively, versus 78.42% for the full model.
- On the AVQA dataset, PSOT surpasses the previous state-of-the-art PSTP by 1.1% average accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PSOT-B instead of PSTP with CLIP-B and VGGish | MUSIC-AVQA expert models · MUSIC-AVQA | Avg. accuracy (%): 73.52 -> 75.29 | Helped |
| Use PSOT-L instead of VALOR | MUSIC-AVQA models · MUSIC-AVQA | Accuracy (%): 78.90 -> 78.42 | Hurt |
| Add M-KPT (ablation) | PSOT · MUSIC-AVQA | Avg. accuracy (%): 77.65 -> 78.42 | Helped |
| Add S-KPT (ablation) | PSOT · MUSIC-AVQA | Avg. accuracy (%): 77.79 -> 78.42 | Helped |
| Add Q-KPT (ablation) | PSOT · MUSIC-AVQA | Avg. accuracy (%): 77.57 -> 78.42 | Helped |
| Add M-KPT and S-KPT together (ablation) | PSOT · MUSIC-AVQA | Avg. accuracy (%): 77.05 -> 78.42 | Helped |
| Use motion-driven adjacency instead of vanilla feature-similarity adjacency (ablation) | M-KPT · MUSIC-AVQA | Avg. accuracy (%): 77.92 -> 78.42 | Helped |
| Use sound-driven adjacency instead of vanilla feature-similarity adjacency (ablation) | S-KPT · MUSIC-AVQA | Avg. accuracy (%): 78.08 -> 78.42 | Helped |
| Execute M-KPT and S-KPT in parallel instead of in series (ablation) | PSOT · MUSIC-AVQA | Avg. accuracy (%): 78.20 / 78.12 -> 78.42 | Helped |
| Use local and global motion together (λ=0.2) rather than local-only or global-only motion (ablation) | M-KPT · MUSIC-AVQA | Avg. accuracy (%): 78.30 / 78.23 -> 78.42 | Helped |
| Use feature similarity rather than optical flow to estimate motion (ablation) | M-KPT · MUSIC-AVQA | Avg. accuracy (%): 78.31 -> 78.42 | Helped |
| Reserve only question-relevant patches rather than all patches (r=80% rather than r=100%) (ablation) | Q-KPT · MUSIC-AVQA | Avg. accuracy (%): 77.30 -> 78.42 | Helped |
Surprising findings
- Using only 20% of visual patches in Q-KPT (77.83%) still outperforms the model keeping 100% of patches (77.30%).
- Replacing optical flow with cosine feature similarity for motion estimation gives slightly higher accuracy (78.42% vs 78.31%) while being faster.
Limitations
- Relies on sampled frames (one per 6 seconds, T=10) to limit computation, which may miss fine-grained temporal changes.
- Motion cues may be less informative when sounding objects are not salient in motion; the authors rely on other modules to compensate.
Experimental setup
Datasets: MUSIC-AVQA AVQA
Benchmarks: MUSIC-AVQA AVQA
Metrics: answer accuracy (%)
Baselines: AVSD PanoAVQA AVST COCA TJSTG AV-Mamba SH-Mamba APL PSTP LAVISH CMP AVLLM
Builds on: CLIP CLAP PSTP
Closest papers by meaning
- Question-Aware Gaussian Experts for Audio-Visual Question Answering (CVPR 2025 · similarity 0.88)
- Object-Aware Adaptive-Positivity Learning for Audio-Visual Question Answering (AAAI 2024 · similarity 0.87)
- Audio-Visual Adaptive Fusion Network for Question Answering Based on Contrastive Learning (AAAI 2025 · similarity 0.87)
- OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.86)
- AVQACL: A Novel Benchmark for Audio-Visual Question Answering Continual Learning (CVPR 2025 · similarity 0.86)
- Look, Listen, and Answer: Overcoming Biases for Audio-Visual Question Answering (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.