Research map / Video / Action recognition
Condensing Action Segmentation Datasets via Generative Network Inversion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the first dataset condensation method for temporal action segmentation, combining a conditional VAE (TCA) generative prior with network inversion to compress segments into latent codes, plus edit-distance-based diverse sequence sampling.
Problem
Storage and processing costs of long procedural video datasets used for temporal action segmentation are large, and existing video dataset condensation methods (for action recognition) cannot handle varying segment lengths, frame-level outputs, or action-sequence redundancy.
Approach
Train a TCA conditional VAE on frame features, then optimize per-segment latent codes via network inversion (decoder frozen) to condense segments along temporal and channel axes; reduce sequence redundancy with furthest-point sampling under normalized edit distance.
Key results
- On Breakfast, storage reduced to 44 MB vs 28 GB original (over 500x) while retaining about 83% of original performance (MS-TCN averaged metrics per paper abstract).
- On Breakfast 10-task incremental setup, the method improves final frame-wise accuracy by 6.8% with MS-TCN and 10.7% with ASFormer over TCA.
- On 50Salads, network inversion raises average F1 from 44.7% to 58.8% over the Encoded baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add network inversion to encoded segment-mean latent codes | Encoded dataset condensation · 50Salads | average F1 score: 44.7% to 58.8% (+14.1%) | Helped |
| replace random sequence sampling with diversity-based sequence sampling (ablation) | Ours · 50Salads | Acc: 71.9% vs. 74.4% | Helped |
| replace random sequence sampling with diversity-based sequence sampling (ablation) | Ours · GTEA | Acc: 74.0 to 75.2 | Helped |
| replace random sequence sampling with diversity-based sequence sampling (ablation) | Ours · Breakfast | Acc: 52.3 to 55.5 | Helped |
| remove sequence sampling (ablation) | Ours · GTEA | Acc: 76.3 vs. 75.2 | Mixed |
| reduce sequence sampling ratio from γ=1 to γ=0.5 (ablation) | Ours · GTEA | Acc: 76.3 to 75.2 | Hurt |
| increase sequence sampling ratio from γ=0.3 to γ=0.4 (ablation) | Ours · GTEA | Acc: 56.9 to 73.6 (16.7%) | Helped |
| increase instances per segment from K=1 to K=8 (ablation) | Ours · Breakfast | Edit: 32.2% to 52.3% | Helped |
| increase instances per segment from K=8 to K=16 (ablation) | Ours · Breakfast | Edit: 52.3 to 51.2 | No clear effect |
| use per-frame latent codes instead of K=8 instances per segment (ablation) | Ours · Breakfast | Edit: 52.3% to 53.3% | No clear effect |
| condense replay data with Ours instead of the baseline | Incremental TAS · Breakfast, 10-task incremental | Acc: 36.0 to 46.7 (10.7%) | Helped |
| condense replay data with Ours instead of the baseline | Incremental TAS · Breakfast, 10-task incremental | Acc: 31.4 to 38.2 | Helped |
Surprising findings
- Boring videos built from Mean or Coreset features can reach up to 80% of full-data performance, revealing strong temporal redundancy in TAS videos.
- Using only half the sequences (diversity sampling) barely hurts performance, indicating substantial sequence redundancy.
Limitations
- Performance plateaus at larger K, suggesting the generative model's expressiveness is a bottleneck.
- Still a gap to full-dataset performance, e.g. about 83% retained on Breakfast and lower Edit/F1 on several settings.
- Evaluated only on three TAS benchmarks with I3D features and two segmentation backbones.
Experimental setup
Datasets: GTEA 50Salads Breakfast
Benchmarks: GTEA 50Salads Breakfast
Metrics: frame-wise accuracy segmental edit score F1@{10,25,50} storage size
Baselines: Mean Coreset TCA Encoded Encoded† Random sampling
Builds on: TCA (temporally coherent action model) dataset condensation furthest point sampling MS-TCN ASFormer
Closest papers by meaning
- Coherent Temporal Synthesis for Incremental Action Segmentation (CVPR 2024 · similarity 0.87)
- ActFusion: a Unified Diffusion Model for Action Segmentation and Anticipation (NeurIPS 2024 · similarity 0.86)
- Efficient and Effective Weakly-Supervised Action Segmentation via Action-Transition-Aware Boundary Alignment (CVPR 2024 · similarity 0.86)
- Unsupervised Action Segmentation via Fast Learning of Semantically Consistent Actoms (AAAI 2024 · similarity 0.85)
- FACT: Frame-Action Cross-Attention Temporal Modeling for Efficient Action Segmentation (CVPR 2024 · similarity 0.85)
- Efficient Temporal Action Segmentation via Boundary-aware Query Voting (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.