Research map / Video / Action recognition
AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AntGPT, which turns recognized video actions into verb-noun text sequences so LLMs can infer goals (top-down) and model action dynamics (bottom-up) for long-term action anticipation, and distills the LLM into a 91M student.
Problem
Long-term action anticipation (LTA) from egocentric and cooking videos requires predicting future verb-noun action sequences; existing bottom-up models lack explicit goal knowledge, and goals are usually unlabeled.
Approach
Represent video observations as sequences of recognized discrete action labels so an off-the-shelf LLM can act as a few-shot goal inferer (top-down planning via in-context learning) and as a temporal dynamics model (bottom-up sequence completion, fine-tuned with LoRA), with knowledge distilled into a small transformer.
Key results
- On Ego4D v2 test, AntGPT reaches verb/noun/action ED@20 of 0.6503/0.6498/0.8770, beating the best listed baseline (PaMsEgoAI 0.6838/0.6785/0.8933).
- On EGTEA Gaze+, AntGPT reaches 80.2 mAP overall and 72.9 on rare actions, versus 76.8 for Anticipatr.
- A 91M-parameter distilled student from Llama2-7B achieves verb/noun ED of 0.6649/0.6752 on Ego4D v2 test, outperforming its 7B teacher (0.6794/0.6757).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add inferred goals from GPT-3.5 to goal-condition the temporal Transformer | Transformer LTA · Ego4D v1 validation | Verb edit distance: 0.735 -> 0.724 | Helped |
| Add inferred goals from GPT-3.5 to goal-condition the temporal Transformer | Transformer LTA · EPIC-Kitchens-55 validation, All | mAP: 38.2 -> 40.1 | Helped |
| Add inferred goals from GPT-3.5 to goal-condition the temporal Transformer | Transformer LTA · EGTEA validation, Rare | mAP: 68.3 -> 72.9 | Helped |
| Use GPT-3-curie as the temporal LLM and add inferred goals (ablation) | GPT-3-curie LTA · Ego4D v1 validation | Verb edit distance: 0.707 -> 0.709 | Hurt |
| Use Llama2-7B as the temporal LLM and add inferred goals (ablation) | Llama2-7B LTA · Ego4D v1 validation | Noun edit distance: 0.705 -> 0.717 | Hurt |
| Replace natural action labels with shuffled semantic labels (ablation) | AntGPT · Ego4D v2 test | Verb edit distance: 0.6794 -> 0.6993 | Hurt |
| Replace natural action labels with dictionary indices (ablation) | AntGPT · Ego4D v2 test | Verb edit distance: 0.6794 -> 0.7249 | Hurt |
| Distill Llama2-7B into a 91M student (ablation) | Llama2-7B temporal model · Ego4D v2 test | Verb edit distance: 0.7176 -> 0.6649 | Helped |
| Use EgoVLP rather than CLIP recognition features (ablation) | AntGPT · Ego4D v2 test | Verb edit distance: 0.6794 -> 0.6677 | Helped |
| Train with both ground-truth and recognized action inputs (ablation) | AntGPT · Ego4D v2 test | Noun edit distance: 0.6721 -> 0.6506 | Helped |
| Increase LLM input context from 1 to 8 observed segments (ablation) | AntGPT · Ego4D v1 validation | Verb edit distance: 0.734 -> 0.707 | Helped |
| Increase autoregressive Transformer depth from 1 to 6 layers (ablation) | Transformer LTA · Ego4D v1 validation | Verb edit distance: 0.739 -> 0.747 | Hurt |
Surprising findings
- Explicit goal inference gives only marginal gains over implicit goal inference when the LLM directly predicts future actions from observed actions.
- Replacing action words with shuffled or index-based labels degrades performance, suggesting LLM language priors contribute to LTA.
Limitations
- Representing video as fixed discrete action labels loses visual detail, limiting tasks beyond anticipation.
- ICL and CoT prompt designs are empirical and performance is sensitive to prompt choices.
- Inferred goal errors propagate and mislead action predictions; pseudo goals rely on metadata or manual annotation.
Experimental setup
Datasets: Ego4D LTA v1 Ego4D LTA v2 EPIC-Kitchens-55 EGTEA Gaze+
Benchmarks: Ego4D LTA v1 Ego4D LTA v2 EPIC-Kitchens-55 EGTEA Gaze+
Metrics: edit distance (ED@20) mean average precision (mAP)
Baselines: HierVL ICVAE VCLIP SlowFast VideoLLM PaMsEgoAI Palm EGO-TOPO Anticipatr Timeception
Builds on: CLIP LoRA Llama 2 Ego-Topo chain-of-thought prompting
Closest papers by meaning
- Can't Make an Omelette Without Breaking Some Eggs: Plausible Action Anticipation Using Large Video-Language Models (CVPR 2024 · similarity 0.89)
- World–Value–Action Model: Implicit Planning for Vision–Language–Action Systems (NeurIPS 2026 · similarity 0.87)
- Towards Understanding Future: Consistency Guided Probabilistic Modeling for Action Anticipation (AAAI 2024 · similarity 0.87)
- Uncertainty-aware Action Decoupling Transformer for Action Anticipation (CVPR 2024 · similarity 0.87)
- Latent Action Reparameterization for Efficient Agent Inference (NeurIPS 2026 · similarity 0.87)
- PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.