Research map / Video / Video-language understanding
From Failure Taxonomy to Intervention: A Diagnostic Methodology for Industry-Scale AVLM in Video and Live-Streaming Platform Moderation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a diagnostic methodology that maps failures of an industrial audio-visual-language model for video moderation into a taxonomy of failure signatures, each linked to a development intervention, and validates it on an in-house 2B AVLM.
Problem
Platform AI moderation failures rarely reveal their cause, and public multimodal foundation-model studies give little systematic guidance on localizing failures and turning them into targeted model-development interventions.
Approach
A failure taxonomy for industrial AVLM development in which each observed failure class (selection-induced support erosion, regime-limited alignment, proxy-sufficient representation collapse, perception-for-reasoning trade-off, perception-moderation mismatch) is paired with a concrete intervention, instantiated via a three-stage audio-visual training pipeline and a cross-modal continuation objecti
Key results
- Cross-modal continuation raised MMSU from 18.98 to 40.98 and MMAU from 34.2 to 52.5 while lowering FLEURS overall WER/CER from 29.26 to 17.98.
- On downstream moderation, full audio adaptation (A+T full) raised Anon.S PR-AUC from 0.447 (ASR+T) to 0.567.
- AVLM-2B achieved FLEURS average WER/CER 13.45 vs 41.15 for Qwen2.5-Omni-3B and 16.03 for Gemma-4-E2B-it, but trailed Qwen2.5-Omni-3B on AVHBench (61.60 vs 71.00) and MMAU (68.30 vs 72.80).
Surprising findings
- A perception-heavy recipe achieved better ASR (LS-Clean 7.21, FLEURS 20.29) yet worse downstream moderation (Anon.S 0.459) than a reasoning-balanced recipe with worse ASR.
- Instruction tuning improved AQA reasoning (MMAU 34.2 to 71.0) while degrading FLEURS WER/CER (29.26 to 34.90).
Limitations
- Production datasets, policy definitions, and moderation examples cannot be released; downstream results use anonymized categories.
- Downstream moderation evaluation uses in-house data only, not reproducible externally.
- Qwen2.5-Omni-3B remains stronger on several audio-reasoning and video benchmarks.
Experimental setup
Datasets: in-house weakly supervised audio-text (ASR pseudo-labels) in-house supervised audio data in-house audio-visual instruction data LibriSpeech FLEURS MMAU MMSU
Benchmarks: AVHBench OmniBench FLEURS MMAU MMSU MMMU RealWorldQA MMStar GQA SEEDBench POPE ScienceQA
Metrics: accuracy WER/CER PR-AUC ANLS F1
Baselines: Qwen2.5-Omni-3B Gemma-4-E2B-it ASR+T transcript-based pipeline
Builds on: visual instruction tuning Qwen2.5-Omni
Closest papers by meaning
- Context-Aware Testing: A New Paradigm for Model Testing with Large Language Models (NeurIPS 2024 · similarity 0.82)
- T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Models (NeurIPS 2024 · similarity 0.82)
- Training ML Models with Predictable Failures (NeurIPS 2026 · similarity 0.82)
- Why Do Multi-Agent LLM Systems Fail? (NeurIPS 2025 · similarity 0.82)
- GUITAR: Structured Failure Diagnosis of GUI Agents via State Transitions (NeurIPS 2026 · similarity 0.81)
- MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers (NeurIPS 2026 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.