Research map / Video / Action recognition
FACT: Frame-Action Cross-Attention Temporal Modeling for Efficient Action Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FACT, a two-branch network (convolutional frame branch and transformer action-token branch) with bidirectional cross-attention and a matching loss for efficient supervised action segmentation, optionally using transcripts to initialize action tokens.
Problem
Supervised action segmentation requires capturing long-range temporal dependencies, but transformer-based frame methods are computationally costly and two-stage methods lack bidirectional transfer between frame and action features.
Approach
Run a convolutional frame branch and a transformer action-token branch in parallel, linked by iterative frame-to-action and action-to-frame cross-attention, with a segment-token matching loss (one-to-one or one-to-many) so each action token encodes a segment; transcripts can initialize tokens.
Key results
- On Breakfast, FACT reaches F1@50 of 66.2 without transcripts versus 64.6 for DiffAct (C1-constrained), and FACT is about 3.5x faster than DiffAct at inference.
- With transcripts, FACT improves F1@50 by 4.5% on Breakfast, 1.5% on GTEA, 6.1% on EgoProceL and 20.1% on EPIC-Kitchen over prior methods.
- FACT uses 0.7 fewer GFLOP-type cost: 5.5 inference GFLOP vs 63.0 for DiffAct on a 5-minute video, i.e. about 91% less than DiffAct.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FACT instead of DiffAct | Action segmentation · Breakfast | F1@50: 64.6 -> 66.2 | Helped |
| Use FACT instead of DiffAct | Action segmentation · EgoProceL | F1@50: 54.6 -> 60.8 | Helped |
| Use FACT instead of LTContext | Action segmentation · EPIC-Kitchen | F1@50: 11.5 -> 18.5 | Helped |
| Initialize action tokens with video transcripts | FACT · Breakfast | F1@50: 69.2 -> 73.7 | Helped |
| Initialize action tokens with video transcripts | FACT · GTEA | F1@50: 86.0 -> 87.5 | Helped |
| Remove action-branch supervision (ablation) | FACT · Breakfast split 1 | F1@50: 65.5 -> 40.0 | Hurt |
| Remove cross-attention supervision (ablation) | FACT · Breakfast split 1 | F1@50: 65.5 -> 43.2 | Hurt |
| Remove frame-branch supervision (ablation) | FACT · Breakfast split 1 | F1@50: 65.5 -> 57.3 | Hurt |
| Remove temporal smoothing loss (ablation) | FACT · Breakfast split 1 | F1@50: 65.5 -> 60.7 | Hurt |
| Use action tokens rather than no action tokens (ablation) | FACT on EgoProceL · EgoProceL | F1@50: +4-6% | Helped |
| Use one-to-one rather than one-to-many token-segment matching (ablation) | FACT on EgoProceL · EgoProceL | F1@50 | Helped |
| Use transcripts during training and testing | FACT with 25% training data · Breakfast | F1@50: 58.4 | Helped |
Surprising findings
- UVAST and DiffAct fail to converge well on EPIC-Kitchen, with very low edit and F1 scores.
- Using 25% of training data with transcripts outperforms the model without transcripts trained on 100% of the data on Breakfast.
Limitations
- Relies on pre-extracted frame features rather than end-to-end training.
- One-to-one matching needs many action tokens on datasets with many segments, so one-to-many matching is used on EPIC-Kitchen.
- Cross-attention cost grows with very long videos, requiring optional temporal downsampling.
Experimental setup
Datasets: Breakfast GTEA EgoProceL EPIC-Kitchen
Benchmarks: Breakfast GTEA EgoProceL EPIC-Kitchen
Metrics: F1@10 F1@25 F1@50 Edit Acc AccB
Baselines: MSTCN MSTCN++ ASFormer UVAST LTContext DiffAct ASRF MuCon RTK TCTr
Builds on: MS-TCN ASFormer DETR UVAST DiffAct
Closest papers by meaning
- Iterative Contrast-Classify for Semi-supervised Temporal Action Segmentation (AAAI 2022 · similarity 0.90)
- Efficient and Effective Weakly-Supervised Action Segmentation via Action-Transition-Aware Boundary Alignment (CVPR 2024 · similarity 0.90)
- Unsupervised Action Segmentation via Fast Learning of Semantically Consistent Actoms (AAAI 2024 · similarity 0.89)
- SCT: Set Constrained Temporal Transformer for Set Supervised Action Segmentation (CVPR 2020 · similarity 0.89)
- Action Shuffle Alternating Learning for Unsupervised Action Segmentation (CVPR 2021 · similarity 0.89)
- OnlineTAS: An Online Baseline for Temporal Action Segmentation (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.