Research map / Video / Action recognition
Understanding Multi-Task Activities from Single-Task Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MT-TAS, training temporal action segmentation models on single-task egocentric videos via LLM-guided synthetic multi-task blending, boundary feature reconstruction, foreground-background feature composition, and foreground-aware refinement, plus a new 12-hour MEKA benchmark.
Problem
Temporal action segmentation models trained on single-task videos fail to handle interleaved steps, task switches, and cluttered multi-task scenes, and no suitable multi-task training or evaluation data exists.
Approach
Synthesize multi-task training videos from single-task videos using LLM-decided task switches, smooth segment boundaries with a learned feature reconstruction module, augment backgrounds by recomposing foreground and background features from different tasks, and refine predictions with foreground-only (action-relevant object) CLIP features.
Key results
- On MEKA offline, the full framework raises MS-TCN F1@50 from 39.7% to 67.4% and FACT F1@50 from 31.7% to 69.9%.
- On MEKA online with ProTAS, accuracy rises from 49.1% to 67.8% and F1@50 from 12.4% to 36.7%.
- Domain adaptation with unlabeled multi-task videos improves offline MS-TCN F1@50 by 4.8 points (to 72.2%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MSB to single-task training (ablation) | MSTCN · MEKA | F1@50: 39.7 -> 56.0 | Helped |
| Add SBL to MSB (ablation) | MSTCN · MEKA | F1@50: 56.0 -> 58.1 | Helped |
| Add FBFC to MSB+SBL (ablation) | MSTCN · MEKA | F1@50: 58.1 -> 58.8 | Helped |
| Add FAAR to MSB+SBL+FBFC (ablation) | MSTCN · MEKA | F1@50: 58.8 -> 67.4 | Helped |
| Add MSB to single-task training (ablation) | FACT · MEKA | F1@50: 31.7 -> 58.4 | Helped |
| Add SBL to MSB (ablation) | FACT · MEKA | F1@50: 58.4 -> 62.3 | Helped |
| Add FBFC to MSB+SBL (ablation) | FACT · MEKA | F1@50: 62.3 -> 65.5 | Helped |
| Add FAAR to MSB+SBL+FBFC (ablation) | FACT · MEKA | F1@50: 65.5 -> 69.9 | Helped |
| Add MSB to single-task training (ablation) | ProTAS · MEKA | Acc: 49.1 -> 51.1 | Helped |
| Add SBL after MSB (ablation) | ProTAS · MEKA | Acc: 51.1 -> 51.1 | No clear effect |
| Add FBFC to MSB+SBL (ablation) | ProTAS · MEKA | F1@50: 14.6 -> 18.3 | Helped |
| Add FAAR to MSB+SBL+FBFC (ablation) | ProTAS · MEKA | F1@50: 18.3 -> 36.7 | Helped |
Surprising findings
- Background-only frames perform poorly for foreground-aware refinement, while foreground frames outperform full images as FAAR input.
- Foreground-only predictions match base accuracy at alpha=1 but have much lower Edit and F1@50, suggesting a lack of temporal consistency.
Limitations
- Synthetic multi-task data relies on an LLM for task-switch decisions and object lists, which may introduce errors.
- Domain adaptation with unlabeled multi-task videos still lags the labeled multi-task upper bound.
- Online performance remains substantially lower than offline performance.
Experimental setup
Datasets: EgoPER MEKA EGTEA
Benchmarks: MEKA EGTEA
Metrics: Acc Acc-bg Edit F1@10 F1@25 F1@50
Baselines: MS-TCN FACT ProTAS LMT training LST&LMT training
Builds on: MS-TCN FACT ProTAS EgoPER GroundingDINO
Closest papers by meaning
- Unsupervised Action Segmentation via Fast Learning of Semantically Consistent Actoms (AAAI 2024 · similarity 0.87)
- Temporal Segmentation of Fine-gained Semantic Action: A Motion-Centered Figure Skating Dataset (AAAI 2021 · similarity 0.87)
- Progress-Aware Online Action Segmentation for Egocentric Procedural Task Videos (CVPR 2024 · similarity 0.87)
- FACT: Frame-Action Cross-Attention Temporal Modeling for Efficient Action Segmentation (CVPR 2024 · similarity 0.87)
- Efficient and Effective Weakly-Supervised Action Segmentation via Action-Transition-Aware Boundary Alignment (CVPR 2024 · similarity 0.86)
- Action Scene Graphs for Long-Form Understanding of Egocentric Videos (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.