Research map / Vision-language and multimodal / Cross-modal learning
EchoTraffic: Enhancing Traffic Anomaly Understanding with Audio-Visual Insights
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AV-TAU, a 29,865-video audio-visual traffic anomaly dataset with 149,325 Q&A pairs, and EchoTraffic, an audio-visual MLLM with an audio-insight frame selector and dynamic connector trained in two phases.
Problem
Traffic anomaly understanding relies mainly on visual input, missing audio cues (crashes, honking) that occur outside the camera view; existing traffic datasets lack audio and detailed textual annotations.
Approach
Select key frames using spectral flux peaks from the audio track, encode audio and video separately, fuse them with a self-attention-weighted dynamic connector into an LLM (LLaMA 2 with LoRA), and train in two phases (InternVid-10M alignment, then AV-TAU instruction tuning).
Key results
- EchoTraffic overall GPT-Eval 0.8103 vs 0.6384 for the next best (Holmes-VAD) on AV-TAU.
- Overall BLEU 0.2520 and BERTScore 0.9041 for EchoTraffic, the highest among compared models.
- Removing audio information drops overall GPT-Eval from 0.8103 to 0.7190 in the ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add audio information (ablation) | EchoTraffic · AV-TAU test (Overall) | GPT-Eval: 0.7190 -> 0.8103 | Helped |
| Add the Audio-Insight Frame Selector (ablation) | EchoTraffic · AV-TAU test (Overall) | GPT-Eval: 0.7713 -> 0.8103 | Helped |
| Add the Audio-Visual Dynamic Connector (ablation) | EchoTraffic · AV-TAU test (Overall) | GPT-Eval: 0.7946 -> 0.8103 | Helped |
| Use EchoTraffic instead of Holmes-VAD | traffic anomaly understanding · AV-TAU test (Overall) | GPT-Eval: 0.6384 -> 0.8103 | Helped |
Limitations
- Long video understanding is identified as a key open challenge.
- Baselines are general video or video anomaly models, since no prior TAU method exists; no audio-capable baseline beyond VideoLLaMA2 is compared.
Experimental setup
Datasets: AV-TAU InternVid-10M
Benchmarks: AV-TAU
Metrics: BLEU ROUGE MoverScore BERTScore GPT-Eval
Baselines: Video-LLaVA LLaMA-VID TimeChat VideoChat2 InternVideo2 VideoLLaMA2 A-Guardian Holmes-VAD
Builds on: InternVideo2 ImageBind LoRA Holmes-VAD Video-LLaMA2
Closest papers by meaning
- Holmes-VAU: Towards Long-term Video Anomaly Understanding at Any Granularity (CVPR 2025 · similarity 0.84)
- TraffiDent: A Dataset for Understanding the Interplay Between Traffic Dynamics and Incidents (NeurIPS 2025 · similarity 0.83)
- TUMTraf VideoQA: Dataset and Benchmark for Unified Spatio-Temporal Video Understanding in Traffic Scenes (ICML 2025 · similarity 0.83)
- FT-AED: Benchmark Dataset for Early Freeway Traffic Anomalous Event Detection (NeurIPS 2024 · similarity 0.83)
- MoniTor: Exploiting Large Language Models with Instruction for Online Video Anomaly Detection (NeurIPS 2025 · similarity 0.82)
- SDAC: A Multimodal Synthetic Dataset for Anomaly and Corner Case Detection in Autonomous Driving (AAAI 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.