Research map / Video / Action recognition
Weakly Supervised Temporal Action Localization via Dual-Prior Collaborative Learning Guided by Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MLLM4WTAL, which uses MLLM-generated key and complete semantic descriptions as priors (via Key Semantic Matching and Complete Semantic Reconstruction modules, linked by dual prior interactive distillation) to improve weakly supervised temporal action localization, with MLLM needed only in training.
Problem
Weakly supervised temporal action localization (WTAL) using only video-level labels often yields incomplete localization (missed sub-actions) and over-complete localization (background included as foreground), while MLLMs are too costly to run at inference.
Approach
Use an MLLM offline during training to produce key action descriptions and complete action descriptions as semantic priors; a key semantic matching branch and a masked-word complete semantic reconstruction branch mutually distill their attention maps, so that the lightweight WTAL localization head runs without the MLLM at inference.
Key results
- On THUMOS14, MLLM4WTAL applied to Zhou et al. reaches 49.8% average mAP (0.1:0.7), vs 48.3% for the base method.
- On ActivityNet-v1.2, MLLM4WTAL (DDG-Net) reaches 30.1% average mAP, versus 27.0% for DDG-Net.
- Ablation on THUMOS14: baseline DELU-style 37.4% average mAP rises to 39.2% with both KSM and CSR; plug-in to UM and CO2-Net improves average mAP by 2.3% and 2.2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add KSM and CSR to Zhou et al. | Zhou et al. · THUMOS14 | average mAP@IoU (0.1:0.7): 48.3 -> 49.8 | Helped |
| Add KSM and CSR to DELU | DELU · THUMOS14 | average mAP@IoU (0.1:0.7): 46.4 -> 47.9 | Helped |
| Add KSM and CSR to DDG-Net | DDG-Net · ActivityNet-v1.2 validation | average mAP: 27.0 -> 30.1 | Helped |
| Add KSM and CSR to DELU | DELU · ActivityNet-v1.2 validation | average mAP: 26.8 -> 28.9 | Helped |
| Add KSM (ablation) | Baseline · THUMOS14 | average mAP@IoU: 37.4 -> 38.9 | Helped |
| Add CSR (ablation) | Baseline · THUMOS14 | average mAP@IoU: 37.4 -> 38.7 | Helped |
| Add both KSM and CSR (ablation) | Baseline · THUMOS14 | average mAP@IoU: 37.4 -> 39.2 | Helped |
| Replace the MLLM key semantic prior with action category names (ablation) | MLLM4WTAL · THUMOS14 | average mAP@IoU: 39.2 -> 37.7 | Hurt |
| Replace the MLLM complete semantic prior with fixed-template action descriptions (ablation) | MLLM4WTAL · THUMOS14 | average mAP@IoU: 39.2 -> 37.6 | Hurt |
| Add KSM, then CSR (ablation) | UM · THUMOS14 | average mAP@IoU: 30.3 -> 32.1 -> 32.6 | Helped |
| Add KSM, then CSR (ablation) | CO2-Net · THUMOS14 | average mAP@IoU: 36.0 -> 37.9 -> 38.2 | Helped |
Limitations
- MLLM is required during training to produce priors, adding training cost.
- Gains are modest (around 1-2 mAP points on average) across the reported settings.
- Key and complete descriptions require known action categories in the prompt.
Experimental setup
Datasets: THUMOS14 ActivityNet-v1.2
Benchmarks: THUMOS14 ActivityNet-v1.2
Metrics: mAP@IoU average mAP
Baselines: CoLA DCC PivoTAL AHLM CASE ISSF DELU Zhou et al. UM CO2-Net DDG-Net
Builds on: UntrimmedNet CO2-Net PivoTAL DELU GloVe
Closest papers by meaning
- Structured Video-Language Modeling with Temporal Grouping and Spatial Grounding (ICLR 2024 · similarity 0.89)
- Similar Modality Enhancement and Action Consistency Learning for Weakly Supervised Temporal Action Localization (AAAI 2025 · similarity 0.89)
- Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders (NeurIPS 2025 · similarity 0.89)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.89)
- Masked Diffusion Vision-Language Models for Temporal Action Localization (NeurIPS 2026 · similarity 0.89)
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.