Research map / Vision-language and multimodal / Cross-modal learning
Query-centric Audio-Visual Cognition Network for Moment Retrieval, Segmentation and Step-Captioning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QUAG, a query-centric audio-visual cognition network that adds modality-synergistic perception and query-centric temporal-channel filtration to the HIREST joint model for moment retrieval, moment segmentation, and step-captioning, and tests on TVSum query-based summarization.
Problem
Learning comprehensive, user-preferred cognition of video content for hierarchical retrieval and step-captioning (HIREST), where prior fusion by element-wise sum/multiplication ignores modality hierarchies and cross-modal association.
Approach
Follows a shallow-to-deep principle: contrastive global alignment and cross-attention local interaction between visual and audio features, then query-guided temporal-channel filtration of the shallow audio-visual representation before a multi-task prediction and captioning pipeline.
Key results
- On HIREST moment retrieval, QUAG reaches 72.54 R@0.5 and 38.86 R@0.7, versus 70.98 and 37.31 for the Joint model.
- On step-captioning, QUAG achieves CIDEr 25.44 and SPICE 4.49 versus 21.19 and 3.02 for Joint, and entailment 40.10% versus 35.97%.
- On TVSum, QUAG averages 87.0 top-5 mAP, on par with TR-DETR (87.1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Joint with QUAG | Moment retrieval · HIREST test | Recall @ IoU 0.5: 70.98 -> 72.54 | Helped |
| Replace Joint with QUAG | Moment retrieval · HIREST test | Recall @ IoU 0.7: 37.31 -> 38.86 | Helped |
| Replace Joint with QUAG | Moment segmentation · HIREST test | Recall @ IoU 0.5: 36.23 -> 39.27 | Helped |
| Replace Joint with QUAG | Step-captioning · HIREST test | CIDEr: 21.19 -> 25.44 | Helped |
| Replace Joint with QUAG | Step-captioning · HIREST test | METEOR: 4.14 -> 4.76; SWinBERT: 5.94 | Mixed |
| Replace TR-DETR with QUAG | Query-based video summarization · TVSum | Avg. top-5 mAP: 87.1 -> 87.0 | Hurt |
| Add MSP, without QC² (ablation) | Joint step-captioning baseline · HIREST test | CIDEr: 21.19 -> 23.18 | Helped |
| Add MSP, without QC² (ablation) | Joint step-captioning baseline · HIREST test | SPICE: 3.02 -> 3.49 | Helped |
| Add QC², without MSP (ablation) | Joint step-captioning baseline · HIREST test | CIDEr: 21.19 -> 20.43 | Hurt |
| Add QC², without MSP (ablation) | Joint step-captioning baseline · HIREST test | SPICE: 3.02 -> 3.87 | Helped |
| Add both MSP and QC² (ablation) | Joint step-captioning baseline · HIREST test | CIDEr: 21.19 -> 25.44 | Helped |
| Add both MSP and QC² (ablation) | Joint step-captioning baseline · HIREST test | SPICE: 3.02 -> 4.49 | Helped |
Surprising findings
- Query-centric filtration alone reduces CIDEr relative to the Joint baseline (20.43 vs 21.19), with gains appearing only when combined with MSP.
Limitations
- Gains over task-specific models (QD-DETR, UVCOM, TR-DETR) are limited on several metrics, especially step-captioning and TVSum.
- Ablation of modules shows modest individual improvements; full gains require both modules.
- Audio is represented only via speech transcription (Whisper), which may miss non-speech sound.
Experimental setup
Datasets: HIREST TVSum
Benchmarks: HIREST TVSum
Metrics: Recall@IoU 0.5 Recall@IoU 0.7 Precision@IoU METEOR ROUGE-L CIDEr SPICE Entailment top-5 mAP
Baselines: Joint QD-DETR UVCOM TR-DETR EVA-CLIP-G/14 BMT SwinBERT TGG UMT
Builds on: HIREST Joint model (Zala et al.) CLIP4Caption InfoNCE
Closest papers by meaning
- JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation (ICLR 2026 · similarity 0.87)
- Fewer Steps, Better Performance: Efficient Cross-Modal Clip Trimming for Video Moment Retrieval Using Language (AAAI 2024 · similarity 0.87)
- Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding (AAAI 2025 · similarity 0.86)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.86)
- Watch and Listen: Understanding Audio-Visual-Speech Moments with Multimodal LLM (NeurIPS 2025 · similarity 0.86)
- PreFM: Online Audio-Visual Event Parsing via Predictive Future Modeling (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.