Research map / Vision-language and multimodal / Robot action and navigation

Masked Diffusion Vision-Language Models for Temporal Action Localization

NeurIPS 2026 · Vision-language and multimodal › Robot action and navigation · approach: Vision-language-action foundation models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Adapts masked diffusion vision-language models (LLaDA-8B backbone) to temporal action localization, using a boundary-aware planned training objective and a step-level soft IoU reward to jointly refine semantic and time tokens during denoising.

Problem

Autoregressive vision-language decoders for temporal action localization generate tokens left to right, so later semantic evidence cannot revise earlier timestamp predictions; standard masked diffusion training also ignores the late reveal of time tokens and uses token-level losses that do not refle

Approach

Treat TAL as masked diffusion sequence generation with bidirectional denoising, plus (1) boundary-aware masking with step-weighted reconstruction so time tokens are masked longer than semantic tokens, and (2) a gated, step-level soft tIoU reward computed from start/end token distributions during denoising.

Key results

• On THUMOS-14, MDVLM-TAL-8B reaches 72.7 average mAP vs. 69.9 for CLTDR-GMG (best listed prior detector).
• On ActivityNet-RTL, MDVLM-TAL achieves 31.24 mIoU and 48.9 Score, versus 28.6 mIoU and 46.3 Score for LITA-13B.
• Ablation on ActivityNet-RTL shows the full model (31.24 mIoU) improves over the base MDVLM (26.94 mIoU).

Surprising findings

• Revealing time tokens earlier in training (PT time) gives a smaller gain than the reasoning-first schedule, supporting late boundary reveal.

Limitations

• Gains on ActivityNet-1.3 are modest relative to some detectors (37.9 avg vs. 37.1 for CLTDR-GMG) and the full table reports only some metrics for some baselines.
• The step-level reward and planned masking add multiple hyperparameters (gamma, eta, mu, delta, lambda_rel) tuned by local sweeps only on loss weights.

Experimental setup

Datasets: ActivityNet-1.3 THUMOS-14 ActivityNet-RTL

Benchmarks: THUMOS-14 ActivityNet-1.3 ActivityNet-RTL

Metrics: mAP at tIoU thresholds mIoU P@0.5 P@0.75 P@0.95 LITA Score

Baselines: ActionFormer TriDet CLTDR-GMG DiffTAD LITA-7B LITA-13B Video-LLaMA-v2 Video-ChatGPT TALLFormer ReAct

Builds on: LLaDA LITA DiffTAD LoRA SigLIP

Closest papers by meaning

• Weakly Supervised Temporal Action Localization via Dual-Prior Collaborative Learning Guided by Multimodal Large Language Models (CVPR 2025 · similarity 0.89)
• Mitigating Mask Prior Drift and Positional Attention Collapse in Large Diffusion Vision-Language Models (ICML 2026 · similarity 0.88)
• ReVisionLLM: Recursive Vision-Language Model for Temporal Grounding in Hour-Long Videos (CVPR 2025 · similarity 0.88)
• Before It Fades: Reinforcing Temporal Representations at Inference Time in VideoLLMs (NeurIPS 2026 · similarity 0.88)
• Bridge the Gap: From Weak to Full Supervision for Temporal Action Localization with PseudoFormer (CVPR 2025 · similarity 0.88)
• Boundary-Aware Temporal Dynamic Pseudo-Supervision Pairs Generation for Zero-Shot Natural Language Video Localization (AAAI 2025 · similarity 0.87)