Research map / Image generation / Autoregressive and latent generation
Action Detection via an Image Diffusion Process
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Recasts temporal action detection as generating three AD images (start, end, action-class) with a discrete action-detection diffusion process and a Row-Column Transformer denoiser.
Problem
Temporal action detection: localizing starting and ending points of action instances in untrimmed videos and classifying them, which existing proposal-based pipelines handle with many hand-designed components.
Approach
Formulates action detection outputs as three grayscale images (N×C action-class, N×2 start and end) and generates them with a diffusion model whose forward process diffuses rows toward a uniform discrete distribution via multinomial noise instead of Gaussian noise, plus a Row-Column Transformer that applies attention across classes and temporal convolution plus attention across frames.
Key results
- Achieves 70.8 average mAP on THUMOS14 (tIoU 0.3-0.7), above TriDet's 69.3 and DiffTAD's 68.0.
- On ActivityNet-1.3 with R(2+1)D features, reports 38.3 average mAP versus TriDet's 36.8.
- Replacing discrete diffusion with standard Gaussian diffusion lowers THUMOS14 average mAP from 69.0 to 66.0 in the same architecture (ablation).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ADI-Diff instead of TriDet | action detection · THUMOS14 test | Avg mAP: 69.3 -> 70.8 | Helped |
| use ADI-Diff instead of TriDet | action detection · ActivityNet-1.3 val | Avg mAP: 36.8 -> 38.3 | Helped |
| replace standard diffusion with Discrete Action-Detection Diffusion (ablation) | ADI-Diff with architecture from [22] · THUMOS14 | Avg mAP: 66.0 -> 69.0 | Helped |
| replace standard diffusion with Discrete Action-Detection Diffusion (ablation) | ADI-Diff with Row-Column Transformer · THUMOS14 | Avg mAP: 68.1 -> 70.8 | Helped |
| use Row-Column Transformer instead of architecture from [22] (ablation) | standard diffusion · THUMOS14 | Avg mAP: 66.0 -> 68.1 | Helped |
| use Row-Column Transformer instead of architecture from [22] (ablation) | Discrete Action-Detection Diffusion · THUMOS14 | Avg mAP: 69.0 -> 70.8 | Helped |
| replace architecture from [22] with architecture from [46] (ablation) | Discrete Action-Detection Diffusion · THUMOS14 | Avg mAP: 67.5 -> 70.8 | Hurt |
| use column design for both row and column processing (ablation) | Row-Column Transformer · THUMOS14 | Avg mAP: 69.0 -> 70.8 | Hurt |
| use row design for both row and column processing (ablation) | Row-Column Transformer · THUMOS14 | Avg mAP: 68.8 -> 70.8 | Hurt |
| replace learnable positional embedding with sinusoidal embedding (ablation) | Row-Column Transformer · THUMOS14 | Avg mAP: 69.1 -> 70.8 | Hurt |
| stitch the three AD images for combined processing (ablation) | ADI-Diff · THUMOS14 | Avg mAP: 70.3 -> 70.8 | Helped |
| stitch the three AD images for combined processing (ablation) | ADI-Diff · THUMOS14 | Speed (seconds per clip): 0.158 -> 0.113 | Helped |
Limitations
- Relies on frozen off-the-shelf video features (I3D, R(2+1)D) and does not train the feature extractor.
- Requires padding videos to a fixed length of 2,304 frames with masking.
- Post-processing pipeline with thresholds and Soft-NMS is still needed to produce final instances.
Experimental setup
Datasets: THUMOS14 ActivityNet-1.3
Benchmarks: THUMOS14 ActivityNet-1.3
Metrics: mAP at tIoU thresholds average mAP
Baselines: TriDet DiffTAD ActionFormer TAGS AFSD BMN ReAct Self-DETR
Builds on: DDPM AFSD TriDet BMN
Closest papers by meaning
- ActFusion: a Unified Diffusion Model for Action Segmentation and Anticipation (NeurIPS 2024 · similarity 0.86)
- DiffDVC: Accurate Event Detection for Dense Video Captioning via Diffusion Models (AAAI 2025 · similarity 0.86)
- Emergent Temporal Correspondences from Video Diffusion Transformers (NeurIPS 2025 · similarity 0.85)
- Dual Conditioned Motion Diffusion for Pose-Based Video Anomaly Detection (AAAI 2025 · similarity 0.85)
- Video Anomaly Detection with Motion and Appearance Guided Patch Diffusion Model (AAAI 2025 · similarity 0.85)
- Video Diffusion Models are Training-free Motion Interpreter and Controller (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.