Research map / Video / Action recognition
Hierarchical Vector Quantization for Unsupervised Action Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Hierarchical Vector Quantization (HVQ), a two-level codebook auto-encoder for unsupervised temporal action segmentation, plus a Jensen-Shannon-distance metric for segment-length bias.
Problem
Unsupervised temporal action segmentation across long untrimmed videos struggles with large intra-class variation in segment lengths; prior optimal-transport pseudo-label methods (TOT, ASOT) show strong segment-length bias and miss short segments.
Approach
Reformulates unsupervised segmentation as joint vector quantization: a frame encoder feeds a fine codebook Z (alpha*K subactions) that maps to a coarse codebook Q (K actions), trained with reconstruction and commitment losses, so each action cluster can be covered by a variable number of subclusters; also introduces a JSD-based segment-length bias metric.
Key results
- On Breakfast, HVQ reaches F1 39.7 and recall 44.9 with JSD 82.5, the best F1, recall and JSD among compared methods, while ASOT has higher MoF (56.1 vs 54.4 for the HVQ variant reported in the ablation)
- On YouTube Instructional, HVQ ties ASOT at F1 35.1 and achieves recall 38.7 vs ASOT 27.8
- On IKEA ASM, HVQ reports MoF 51.2, F1 30.7, JSD 64.8, outperforming CTE, TOT, TOT+TCL and ASOT on all metrics
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use hierarchical vector quantization instead of single-level quantization (ablation) | HVQ · Breakfast | F1: 37.1 -> 39.7 | Helped |
| Use hierarchical vector quantization instead of single-level quantization (ablation) | HVQ · YouTube Instructional | F1: 33.0 -> 35.1 | Helped |
| Use three hierarchy levels instead of two (ablation) | HVQ · IKEA ASM | F1: 27.6 -> 30.7 | Helped |
| Use three hierarchy levels instead of two (ablation) | HVQ · YouTube Instructional | F1: 35.1 -> 31.9 | Hurt |
| Add reconstruction loss to both commitment losses (ablation) | HVQ · Breakfast | F1: 38.0 -> 39.7 | Helped |
| Use fine-level commitment loss rather than reconstruction loss alone (ablation) | HVQ · Breakfast | F1: 29.7 -> 37.4 | Helped |
| Set hierarchy scale α=2 instead of α=1 (ablation) | HVQ · Breakfast | F1: 38.2 -> 39.7 | Helped |
| Increase hierarchy scale from α=2 to α=3 (ablation) | HVQ · Breakfast | F1: 39.7 -> 38.3 | Hurt |
| Use two MS-TCN stages instead of one (ablation) | HVQ encoder and decoder · Breakfast | F1: 34.4 -> 39.7 | Helped |
| Use MS-TCN instead of an MLP decoder (ablation) | HVQ decoder · Breakfast | F1: 37.5 -> 39.7 | Helped |
| Apply FIFA decoding at inference (ablation) | HVQ · Breakfast | F1: 29.9 -> 39.7 | Helped |
| Use K-Means on autoencoder features instead of joint vector quantization (ablation) | HVQ · Breakfast | F1: 25.4 -> 39.7 | Hurt |
Surprising findings
- ASOT achieves the highest MoF on Breakfast despite much lower recall and higher JSD, showing MoF hides short-segment bias
- Adding a third hierarchy level does not improve Breakfast or YTI results, only F1 on IKEA ASM
Limitations
- Fails to identify some small segments (e.g., small 'pour oil' segment in friedegg example)
- Number of hierarchy levels and alpha must be chosen per dataset based on expected action lengths
- MoF is lower than ASOT on Breakfast and YouTube Instructional
Experimental setup
Datasets: Breakfast YouTube Instructional IKEA ASM
Benchmarks: Breakfast YouTube Instructional IKEA ASM
Metrics: MoF F1 precision recall JSD
Baselines: CTE TOT TOT+TCL ASOT UFSA UDE ASAL JVT Mallow
Builds on: TOT ASOT MS-TCN FIFA VQ-VAE
Closest papers by meaning
- Temporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentation (CVPR 2021 · similarity 0.91)
- Unsupervised Learning of Action Classes With Continuous Temporal Embedding (CVPR 2019 · similarity 0.90)
- Unsupervised Action Segmentation via Fast Learning of Semantically Consistent Actoms (AAAI 2024 · similarity 0.89)
- Temporally Consistent Unbalanced Optimal Transport for Unsupervised Action Segmentation (CVPR 2024 · similarity 0.88)
- HopaDIFF: Holistic-Partial Aware Fourier Conditioned Diffusion for Referring Human Action Segmentation in Multi-Person Scenarios (NeurIPS 2025 · similarity 0.86)
- FACT: Frame-Action Cross-Attention Temporal Modeling for Efficient Action Segmentation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.