Research map / Video / Action recognition
HopaDIFF: Holistic-Partial Aware Fourier Conditioned Diffusion for Referring Human Action Segmentation in Multi-Person Scenarios
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces referring human action segmentation (RHAS) with the RHAS133 movie dataset and HopaDIFF, a dual-branch diffusion model using HP-xLSTM cross-input-gate fusion and DFT frequency conditioning.
Problem
Action segmentation in untrimmed multi-person videos where a textual reference specifies which person's actions to segment; existing methods assume single-person fixed action protocols.
Approach
A holistic branch and a GroundingDINO-cropped partial branch, both with BLIP-2/CLIP features, are fused via cross-input-gate attentional xLSTM (HP-xLSTM) and conditioned on DFT frequency features within an ActDiff-style diffusion segmenter.
Key results
- On RHAS133 random partition with BLIP-2 (test), HopaDIFF reaches 62.58% ACC, 7.75 EDIT, 87.96/85.50/79.39 F1@10/25/50, versus best baseline ActDiff at 41.85% ACC and 70.56 F1@10.
- On cross-movie evaluation (test), HopaDIFF gets 59.63% ACC and 90.91/90.33/89.26 F1@10/25/50, versus best baseline LTContent at 52.52% ACC.
- With CLIP features (random partition, test), HopaDIFF reaches 46.62% ACC vs best baseline ASQuery at 26.35% ACC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the GroundingDINO-based partial reasoning branch (ablation) | HopaDIFF · RHAS133 cross-movie test | F1@50: 49.42 -> 89.26 | Helped |
| Add HP-xLSTM temporal aggregation (ablation) | HopaDIFF · RHAS133 cross-movie test | F1@50: 83.50 -> 89.26 | Helped |
| Add bidirectional cross-attention fusion between branch input gates (ablation) | HopaDIFF · RHAS133 cross-movie test | F1@50: 81.38 -> 89.26 | Helped |
| Add frequency-domain conditioning (ablation) | HopaDIFF · RHAS133 cross-movie test | F1@50: 86.38 -> 89.26 | Helped |
| Add the proposed components rather than using vanilla diffusion segmentation (ablation) | HopaDIFF · RHAS133 cross-movie test | F1@50: 21.80 -> 89.26 | Helped |
| Use both holistic and partial branches instead of only the partial branch (ablation) | HopaDIFF · RHAS133 cross-movie test | F1@50: 80.48 -> 89.26 | Helped |
| Replace FACT holistic input with a partial branch (ablation) | FACT · RHAS133 cross-movie test | F1@50: 57.13 -> 62.40 | Helped |
| Replace FACT holistic input with a partial branch (ablation) | FACT · RHAS133 cross-movie test | ACC: 27.89 -> 18.34 | Hurt |
| Add DFT conditioning to ActDiff (ablation) | ActDiff · RHAS133 cross-movie test | F1@50: 21.80 -> 28.92 | Helped |
| Add DFT conditioning to ActDiff (ablation) | ActDiff · RHAS133 cross-movie validation | F1@50: 37.51 -> 23.40 | Hurt |
| Use HP-xLSTM instead of cross-attention fusion (ablation) | HopaDIFF · RHAS133 cross-movie test | F1@50: 87.36 -> 89.26 | Helped |
| Keep separate branch heads instead of merging them (ablation) | HopaDIFF · RHAS133 cross-movie test | F1@50: 86.64 -> 89.26 | Helped |
Surprising findings
- ActDiff collapses on cross-movie evaluation (2.36% test ACC, F1@10 22.44), suggesting learned action priors do not transfer across movies.
- Baselines and HopaDIFF show very low EDIT scores on several settings (e.g., FACT ~0.27-0.44), while ActDiff gets high EDIT, indicating metric trade-offs.
Limitations
- Experiments limited to the single constructed dataset, constraining generalizability evaluation.
- Two-stage pipeline relying on pre-extracted VLM features limits inference efficiency.
- Performance drops on rare classes, which are often misclassified as frequent classes.
Experimental setup
Datasets: RHAS133
Benchmarks: RHAS133
Metrics: ACC EDIT F1@10 F1@25 F1@50
Baselines: FACT ActDiff ASQuery LTContent RefAtomNet
Builds on: ActDiff xLSTM FACT ASFormer
Closest papers by meaning
- Unsupervised Action Segmentation via Fast Learning of Semantically Consistent Actoms (AAAI 2024 · similarity 0.88)
- ActFusion: a Unified Diffusion Model for Action Segmentation and Anticipation (NeurIPS 2024 · similarity 0.87)
- FACT: Frame-Action Cross-Attention Temporal Modeling for Efficient Action Segmentation (CVPR 2024 · similarity 0.87)
- Hierarchical Vector Quantization for Unsupervised Action Segmentation (AAAI 2025 · similarity 0.86)
- Stitch, Contrast, and Segment: Learning a Human Action Segmentation Model Using Trimmed Skeleton Videos (AAAI 2025 · similarity 0.86)
- FineParser: A Fine-grained Spatio-temporal Action Parser for Human-centric Action Quality Assessment (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.