Research map / Probabilistic and scientific modeling / Molecules and proteins
Uncertainty-aware Action Decoupling Transformer for Action Anticipation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UADT, a two-stream transformer that decouples action anticipation into verb and noun predictions, using epistemic uncertainty from probabilistic encoders to mask embeddings and fuse the streams.
Problem
Human action anticipation predicts future (verb, noun) actions from past video, but jointly predicting both parts makes errors in either compound; existing methods ignore verb-noun dependencies.
Approach
Decouple action anticipation into a verb-to-noun model and a noun-to-verb model that assist each other, quantify predictive epistemic uncertainty via Gaussian probabilistic layers to mask unreliable embeddings, and fuse the two streams by their uncertainties, followed by valid-pair post-processing.
Key results
- On EK100 validation with K700 features, UADT reaches 38.2 verb, 41.4 noun and 20.3 action top-5 recall (RGB), and 23.0 action with RGB+Flow+Obj, beating RAFTformer-2B's 19.1 action.
- On EGTEA Gaze+ with TSN/IN1k features, UADT achieves 68.4% top-5 recall vs. 63.5% for RAFTformer.
- On 50-Salads, UADT achieves 62.7% top-1 accuracy vs. 59.6% for Latent-goal.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UADT instead of RAFTformer | Action anticipation · EGTEA Gaze+ | Top-5 recall: 63.5 -> 68.4 | Helped |
| Use UADT instead of Latent-goal | Action anticipation · 50-Salads | Top-1 accuracy (%): 59.6 -> 62.7 | Helped |
| Add flow and object modalities to RGB (ablation) | UADT · EK100 validation, K700 features | Top-5 recall (action): 20.3 -> 23.0 | Helped |
| Add encoder uncertainty modeling (ablation) | UADT · EK100 validation | Top-5 recall (action): 18.0 -> 18.8; 19.4 -> 20.3 | Helped |
| Use epistemic rather than total or aleatoric uncertainty (ablation) | UADT · EK100 validation | Top-5 recall (action): 18.0 -> 18.8; 19.4 -> 20.3 | Helped |
| Train end-to-end instead of using two-stage training (ablation) | UADT · EK100 validation | Top-5 recall (action): 18.8 -> 19.5; 20.3 -> 21.2 | Helped |
| Use uncertainty-based fusion instead of early fusion (ablation) | UADT · EK100 validation | Top-5 recall (action): 17.9 -> 18.8; 19.2 -> 20.3 | Helped |
Surprising findings
- Ablation shows epistemic uncertainty is more effective than aleatoric or total uncertainty for masking embeddings.
Limitations
- Uncertainty estimation requires repeated stochastic forward passes (25 samples), adding inference cost.
- Verb and noun are predicted separately, so post-processing is needed to remove implausible pairs.
- Evaluated only on short-term anticipation.
Experimental setup
Datasets: EPIC-KITCHENS-100 EGTEA Gaze+ 50-Salads
Benchmarks: EPIC-KITCHENS-100 EGTEA Gaze+ 50-Salads
Metrics: top-5 recall top-1 accuracy
Baselines: RULSTM AVT AVT+ TempAgg MeMViT RAFTformer Latent-goal MCE RED
Builds on: AVT RAFTformer rolling-unrolling LSTM Gaussian probabilistic layers for uncertainty
Closest papers by meaning
- Can't Make an Omelette Without Breaking Some Eggs: Plausible Action Anticipation Using Large Video-Language Models (CVPR 2024 · similarity 0.88)
- AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos? (ICLR 2024 · similarity 0.87)
- Towards Understanding Future: Consistency Guided Probabilistic Modeling for Action Anticipation (AAAI 2024 · similarity 0.85)
- ATI-VLA: Action-Centric Predictive Vision–Language–Action Models via Actionable Alignment Then Adaptive Injection (NeurIPS 2026 · similarity 0.84)
- On Discriminative vs. Generative classifiers: Rethinking MLLMs for Action Understanding (ICLR 2026 · similarity 0.83)
- Grounding Multimodal Large Language Models in Actions (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.