Research map / Video / Video-language understanding
PreFM: Online Audio-Visual Event Parsing via Predictive Future Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces On-AVEP, an online audio-visual event parsing paradigm, and PreFM, a lightweight model that predicts pseudo-future audio-visual context and distills OnePeace text features, reaching SOTA on UnAV-100 and LLP with far fewer parameters.
Problem
Audio-visual event parsing methods process entire videos offline with large models, which is unsuitable for real-time streaming use; online inference lacks future context to disambiguate events.
Approach
Predict pseudo-future audio-visual features from current and past context via a Universal Hybrid Attention block with temporal-modality cross fusion, distill modality-agnostic features from a frozen OnePeace text encoder, and weight losses with a Gaussian focal temporal prior centered on the current step.
Key results
- On UnAV-100 on-AVEL, PreFM reaches 46.3 event-level Avg F1 and 70.1 mAP, +9.3 Avg F1 over the next best offline-style baseline, with 6.5M params versus 238.8M for CCNet.
- PreFM+ reaches 51.5 Avg F1 on UnAV-100 On-AVEL (+14.5 over the next best per the paper).
- On LLP On-AVVP, PreFM achieves 41.2 Avg F1_av and 61.3 mAP_av with 3.3M params and 94.4 FPS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PreFM instead of CCNet | UnAV-100 On-AVEL · UnAV-100 On-AVEL | Avg F1: 37.0 -> 46.3 | Helped |
| Use PreFM instead of CCNet | UnAV-100 On-AVEL · UnAV-100 On-AVEL | mAP: 62.3 -> 70.1 | Helped |
| Use PreFM instead of MM-CSE | LLP On-AVVP · LLP On-AVVP | Avg F1 av: 36.2 -> 41.2 | Helped |
| Add pseudo-future mechanism (ablation) | On-AVEL baseline without proposed components · UnAV-100 On-AVEL | Avg F1: 40.8 -> 42.4 | Helped |
| Add focal temporal prioritization (ablation) | On-AVEL with pseudo-future mechanism · UnAV-100 On-AVEL | Avg F1: 42.4 -> 43.0 | Helped |
| Add modality-agnostic robust representation (ablation) | On-AVEL with pseudo-future mechanism · UnAV-100 On-AVEL | Avg F1: 40.8 -> 44.2 | Helped |
| Add random segment sampling (ablation) | On-AVEL with pseudo-future mechanism · UnAV-100 On-AVEL | Avg F1: 40.8 -> 44.0 | Helped |
| Add all proposed components including random segment sampling (ablation) | On-AVEL baseline without proposed components · UnAV-100 On-AVEL | Avg F1: 40.8 -> 46.3 | Helped |
| Add pseudo-future mechanism without future-oriented losses (ablation) | On-AVEL model without pseudo-future module · UnAV-100 On-AVEL | Avg F1: 44.7 -> 44.8 | No clear effect |
| Add future MRR loss (ablation) | On-AVEL model with pseudo-future mechanism · UnAV-100 On-AVEL | Avg F1: 44.8 -> 45.2 | Helped |
| Add direct future supervision (ablation) | On-AVEL model with pseudo-future mechanism · UnAV-100 On-AVEL | Avg F1: 44.8 -> 45.5 | Helped |
| Use temporal-modality cross fusion instead of self-only, temporal-only, or modality-only fusion (ablation) | PreFM pseudo-future modeling · UnAV-100 On-AVEL | Avg F1: 44.6; 45.3; 45.0 -> 46.3 | Helped |
Surprising findings
- Pseudo-future features reach a Top-5 similarity accuracy above 94% across T+1 to T+5, while Top-1 is only about 42-45%.
- Event-level performance drops sharply in dense scenes: Avg F1 is 0.54 for 1-3 events but 0.13 for more than 6 events.
Limitations
- Designed only for event detection and localization; not extended to QA or captioning.
- Pseudo-future predictions can be noisy and degrade performance.
- Performance degrades in dense scenes with many simultaneous event classes.
Experimental setup
Datasets: UnAV-100 LLP
Benchmarks: UnAV-100 LLP
Metrics: segment-level F1 mAP event-level F1 at tIoU 0.1-0.9 Avg F1 params FLOPs peak memory FPS latency
Baselines: UnAV UniAV CCNet VALOR CoLeaF LEAP NREP MM-CSE
Builds on: UAVM-style audio-visual video parsing UnAV VALOR OnePeace
Closest papers by meaning
- Towards Open-Vocabulary Audio-Visual Event Localization (CVPR 2025 · similarity 0.90)
- Multimodal Class-aware Semantic Enhancement Network for Audio-Visual Video Parsing (AAAI 2025 · similarity 0.90)
- Adapting to the Unknown: Training-Free Audio-Visual Event Perception with Dynamic Thresholds (CVPR 2025 · similarity 0.90)
- Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding (AAAI 2025 · similarity 0.88)
- FAVE: A Structured Benchmark for Fine-Grained Audio-Visual Temporal Evaluation in Multimodal LLMs (CVPR 2026 · similarity 0.88)
- Audio-Visual Semantic Graph Network for Audio-Visual Event Localization (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.