Research map / Video / Video-language understanding
MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the AVHaystacksQA task and AVHaystacks benchmark for multi-video audio-visual retrieval and grounded QA, plus the MAGNET multi-agent RAG framework and two metrics (StEM, MTGS).
Problem
Existing audio-visual video QA benchmarks associate each question with a single short clip, so LMMs are not evaluated on retrieving and reasoning over salient audio-visual segments across large multi-video collections.
Approach
A retrieval-augmented multi-agent pipeline: ImageBind-based video retrieval, a salient frame selector (SFS) that picks diverse, temporally dispersed frames, one AVLLM agent per shortlisted video producing grounded segments, and a meta-agent that synthesizes the final stepwise answer; plus step-wise error and matched temporal grounding metrics.
Key results
- MAGNET+Qwen 2.5 Omni-FT reaches BLEU@4 55.82 and GPT Eval 7.84 on AVHaystacks-50, versus 11.64 BLEU@4 for the best non-agentic baseline (Unified IO2), and 42.64-43.16 for VideoRAG/Video-RAG.
- On AVHaystacks-Full, MAGNET retrieval gets R@5 of 79.20 versus 74.96 for VideoRAG.
- MTGS_avg of MAGNET+Qwen 2.5 Omni-FT is 0.79 on AVHaystacks-Full compared with 0.49 for its zero-shot variant.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MAGNET instead of VideoRAG for retrieval | AVHaystacks · AVHaystacks-50 | R@3: 85.57 -> 90.68 | Helped |
| Use MAGNET instead of VideoRAG for retrieval | AVHaystacks · AVHaystacks-Full | R@5: 74.96 -> 79.20 | Helped |
| Use MAGNET with fine-tuned Qwen 2.5 Omni instead of VideoRAG | AVHaystacksQA · AVHaystacks-50 | BLEU@4: 43.16 -> 55.82 | Helped |
| Use both audio and visual modalities rather than visual only (ablation) | MAGNET + Qwen-2.5-Omni-FT · AVHaystacks-Full | BLEU@4: 45.28 -> 53.64 | Helped |
| Use both audio and visual modalities rather than audio only (ablation) | MAGNET + Qwen-2.5-Omni-FT · AVHaystacks-Full | BLEU@4: 38.96 -> 53.64 | Helped |
| Replace uniform frame sampling with SFS (ablation) | MAGNET + Qwen-2.5-Omni-FT · AVHaystacks-Full | BLEU@4: 36.58 -> 53.61 | Helped |
| Set top-k video selection from 1 to 6 (ablation) | MAGNET + Qwen-2.5-Omni-FT · AVHaystacks-Full | BLEU@4: 46.67 -> 53.61 | Helped |
| Set top-k video selection from 6 to 10 (ablation) | MAGNET + Qwen-2.5-Omni-FT · AVHaystacks-Full | BLEU@4: 53.61 -> 53.38 | Hurt |
| Replace cosine or exponential sampling penalties with sinusoidal-inverse SFS (ablation) | MAGNET + Qwen-2.5-Omni-FT · AVHaystacks-Full | BLEU@4: 52.85; 53.18 -> 53.61 | Helped |
| Increase candidate frame count from 15 to 75 (ablation) | MAGNET + Qwen-2.5-Omni-FT · AVHaystacks-Full | BLEU@4: 45.89 -> 53.61 | Helped |
| Increase text similarity threshold from 0.3 to 0.5 (ablation) | MAGNET + Qwen-2.5-Omni-FT · AVHaystacks-Full | BLEU@4: 51.32 -> 53.61 | Helped |
| Increase text similarity threshold from 0.5 to 0.7 (ablation) | MAGNET + Qwen-2.5-Omni-FT · AVHaystacks-Full | BLEU@4: 53.61 -> 50.67 | Hurt |
Surprising findings
- Plain Qwen 2.5 Omni, Unified IO2 and Video-SALMONN score far below retrieval-augmented baselines (BLEU@4 around 11-12 vs 42-43), showing that long multi-video context is hard without retrieval.
Limitations
- Relies on off-the-shelf components (ImageBind, Gemini captions, GPT-4o meta-agent) rather than end-to-end trainable modules.
- Benchmark is built from GPT-generated questions and answers with limited release statements; the paper points to a supplementary video sample only.
- Scale of the benchmark is modest (500 videos, 3100 QA pairs) and the retrieval-augmented pipeline needs multiple AVLLM agent calls, increasing compute with k.
Experimental setup
Datasets: AVHaystacks
Benchmarks: AVHaystacks-50 AVHaystacks-Full
Metrics: BLEU@4 CIDEr Text Sim GPT Eval Human Eval R@3 R@5 MTGS StEM Recall
Baselines: VideoRAG Video-RAG Qwen2.5 Omni Unified IO2 Video-SALMONN ImageBind-RAG Text-RAG
Builds on: Video-RAG VideoRAG Qwen2.5-Omni ImageBind
Closest papers by meaning
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.90)
- Question-Aware Gaussian Experts for Audio-Visual Question Answering (CVPR 2025 · similarity 0.89)
- Searching Videos as Trees: Self-Correcting Agents for Grounded Long Video QA (NeurIPS 2026 · similarity 0.89)
- CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understanding (ICLR 2025 · similarity 0.89)
- OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.89)
- Modularized Self-Reflected Video Reasoner for Multimodal LLM with Application to Video Question Answering (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.