Research map / Video / Action recognition
Coherent Temporal Synthesis for Incremental Action Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Temporally Coherent Action (TCA), a conditional VAE with a relative-progression coherence variable, to generate replay features for incremental temporal action segmentation on Breakfast and YouTube Instructional.
Problem
Data replay for incremental learning in video has been limited to storing frame exemplars designed for action recognition; temporal action segmentation needs replay that is memory-efficient and temporally coherent across long procedural videos.
Approach
Model each action's frame features with a conditional VAE conditioned on action label and a coherence variable (relative progression within the segment), then generate replay segments top-down from sampled action sequences and durations, with a fixed latent per segment for diversity and smooth progression for coherence.
Key results
- On Breakfast 10-task disjoint setup with MSTCN, the proposed method reaches 29.4% accuracy versus 16.1% for the Exemplar baseline and 7.4% for Finetune.
- With ASFormer on Breakfast 10-task, accuracy is 34.2% versus 12.4% for Exemplar; the original-replay upper bound is 48.1%.
- On YouTube Instructional 5-task, the method performs similarly to Exemplar (30.2% vs 30.8% accuracy with MSTCN), with a less significant gain attributed to lower segment diversity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace fine-tuning-only learning with TCA generated replay | MSTCN · Breakfast, 10 tasks | accuracy: 7.4 -> 29.4 | Helped |
| Replace exemplar replay with TCA generated replay | MSTCN · Breakfast, 10 tasks | accuracy: 16.1 -> 29.4 | Helped |
| Use TCA generated replay rather than exemplar replay | MSTCN · YouTube Instructional, 5 tasks | accuracy: 30.8 -> 30.2 | Hurt |
| Add temporal coherence to generated replay segments (ablation) | TCA · Breakfast | accuracy: 32.9 -> 41.8 | Helped |
| Add within-segment feature diversity (ablation) | TCA · Breakfast | accuracy: 37.9 -> 41.8 | Helped |
| Use generated static action segments instead of frame exemplars (ablation) | TCA · Breakfast | accuracy: 27.8 -> 37.9 | Helped |
| Increase replay size from M=30 to M=60 (ablation) | TCA replay · Breakfast | accuracy: 34.0 -> 35.4 | Helped |
| Increase replay size from M=60 to M=90 (ablation) | TCA replay · Breakfast | accuracy: 35.4 -> 36.2 | Helped |
| Increase replay size from M=90 to M=120 (ablation) | TCA replay · Breakfast | accuracy: 36.2 -> 38.0 | Helped |
| Increase TCA training data from 25% to 50% (ablation) | TCA · Breakfast | accuracy: 41.7 -> 42.1 | Helped |
| Increase TCA training data from 50% to 75% (ablation) | TCA · Breakfast | accuracy: 42.1 -> 45.3 | Helped |
| Increase TCA training data from 75% to 100% (ablation) | TCA · Breakfast | accuracy: 45.3 -> 47.4 | Helped |
Surprising findings
- On YouTube Instructional the proposed generative replay does not clearly beat frame-exemplar replay in frame-wise accuracy, which the authors attribute to reduced segment diversity in that smaller dataset.
- ASFormer is more prone to overfit static replay data than MSTCN, performing worse than MSTCN with the Exemplar baseline.
Limitations
- Requires dense frame-level labels for all videos in each incremental task, which is expensive to obtain.
- Generated features are in a learned feature space and cannot be decoded into human-readable video, making qualitative evaluation hard.
- Performance on YouTube Instructional shows little gain over the exemplar baseline, and a notable gap remains to the original-replay upper bound on Breakfast.
Experimental setup
Datasets: Breakfast YouTube Instructional
Benchmarks: Breakfast YouTube Instructional
Metrics: frame-wise accuracy edit score F1@{10,25,50}
Baselines: Finetune Exemplar Original MSTCN ASFormer
Builds on: MSTCN ASFormer conditional VAE blurry class-incremental learning
Closest papers by meaning
- Condensing Action Segmentation Datasets via Generative Network Inversion (CVPR 2025 · similarity 0.87)
- OnlineTAS: An Online Baseline for Temporal Action Segmentation (NeurIPS 2024 · similarity 0.85)
- FACT: Frame-Action Cross-Attention Temporal Modeling for Efficient Action Segmentation (CVPR 2024 · similarity 0.85)
- Unsupervised Action Segmentation via Fast Learning of Semantically Consistent Actoms (AAAI 2024 · similarity 0.85)
- ActFusion: a Unified Diffusion Model for Action Segmentation and Anticipation (NeurIPS 2024 · similarity 0.85)
- Understanding Multi-Task Activities from Single-Task Videos (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.