Research map / Video / Action recognition
ActFusion: a Unified Diffusion Model for Action Segmentation and Anticipation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ActFusion, a unified diffusion model that jointly performs temporal action segmentation and long-term action anticipation via anticipative and random masking with learnable mask tokens, and reports SOTA on 50 Salads, Breakfast, and GTEA.
Problem
Temporal action segmentation (TAS) and long-term action anticipation (LTA) are studied as separate tasks, and prior unified models generalize poorly across them; LTA evaluations also leak ground-truth future length.
Approach
Train one diffusion encoder-decoder (ASFormer-based) on both visible and invisible frames: anticipative masking replaces late frames with learnable mask tokens so the same model segments observed frames and predicts future actions, combined with random, relation, and boundary masks.
Key results
- On 50 Salads TAS, ActFusion reaches F1@{10,25,50} of 91.6/90.7/84.8, edit 86.0, accuracy 89.3 (avg 88.5), exceeding DiffAct's avg 87.4.
- On Breakfast LTA (alpha=0.2, beta=0.1), ActFusion achieves 28.25 MoC versus FUTR's 27.70 under the ground-truth-length protocol.
- Under the rectified protocol without ground-truth length, ActFusion† reaches 41.14 on 50 Salads (alpha=0.3, beta=0.1) versus FUTR† at 21.52.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Jointly train segmentation and anticipation, including segmentation loss on observed frames (ablation) | ActFusion · 50 Salads LTA (alpha=0.2, beta=0.1) | MoC: 27.32 -> 39.55 | Helped |
| Add anticipative masking (ablation) | ActFusion · 50 Salads TAS | Avg.: 87.2 -> 88.5 | Helped |
| Add anticipative masking (ablation) | ActFusion · 50 Salads LTA (alpha=0.2, beta=0.1) | MoC: 35.24 -> 42.80 | Helped |
| Add random masking (ablation) | ActFusion · 50 Salads TAS | Avg.: 86.8 -> 88.5 | Helped |
| Add boundary masking (ablation) | ActFusion · 50 Salads TAS | Avg.: 87.0 -> 88.5 | Helped |
| Add relation masking (ablation) | ActFusion · 50 Salads TAS | Avg.: 87.2 -> 88.5 | Helped |
| Use learnable mask tokens instead of zero vectors (ablation) | ActFusion · 50 Salads TAS | Avg.: 86.9 -> 88.5 | Helped |
| Place mask tokens in the encoder rather than the decoder (ablation) | ActFusion · 50 Salads TAS | Avg.: 86.7 -> 88.5 | Helped |
| Add relative position bias instead of no position embedding (ablation) | ActFusion · 50 Salads TAS | Avg.: 87.7 -> 88.5 | Helped |
| Add smoothing loss to boundary loss (ablation) | ActFusion · 50 Salads TAS | Avg.: 84.3 -> 88.2 | Helped |
| Add cross-entropy loss to boundary and smoothing losses (ablation) | ActFusion · 50 Salads TAS | Avg.: 88.2 -> 88.5 | Helped |
| Add reconstruction loss (ablation) | ActFusion · 50 Salads TAS | Avg.: 81.3 -> 88.5 | Helped |
Surprising findings
- Task-specific models such as DiffAct and FUTR generalize poorly to the other task, per the cross-task evaluation.
- Previously reported LTA results benefit from ground-truth future length, which leaks information; all compared methods degrade when it is removed.
Limitations
- Relies on pre-extracted I3D features rather than end-to-end video encoders.
- Random masking and reconstruction loss degrade long-term anticipation performance in some settings.
Experimental setup
Datasets: 50 Salads Breakfast GTEA
Benchmarks: 50 Salads Breakfast GTEA
Metrics: F1@{10,25,50} edit score frame-wise accuracy mean over classes accuracy
Baselines: MS-TCN MS-TCN++ SSTDA BCN ASFormer DiffAct FUTR TempAgg Cycle Cons. A-ACT ObjectPrompt UVAST
Builds on: DiffAct ASFormer FUTR DDIM TempAgg
Closest papers by meaning
- Unsupervised Action Segmentation via Fast Learning of Semantically Consistent Actoms (AAAI 2024 · similarity 0.89)
- OnlineTAS: An Online Baseline for Temporal Action Segmentation (NeurIPS 2024 · similarity 0.87)
- HopaDIFF: Holistic-Partial Aware Fourier Conditioned Diffusion for Referring Human Action Segmentation in Multi-Person Scenarios (NeurIPS 2025 · similarity 0.87)
- Masked Temporal Interpolation Diffusion for Procedure Planning in Instructional Videos (ICLR 2025 · similarity 0.86)
- Masked Diffusion Vision-Language Models for Temporal Action Localization (NeurIPS 2026 · similarity 0.86)
- FACT: Frame-Action Cross-Attention Temporal Modeling for Efficient Action Segmentation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.