Research map / Video / Action recognition
Can't Make an Omelette Without Breaking Some Eggs: Plausible Action Anticipation Using Large Video-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PlausiVL, a BLIP-2-based video-LLM fine-tuned with a counterfactual plausible-action-sequence loss and a long-horizon repetition-penalty loss to anticipate plausible future action sequences on Ego4D and EPIC-Kitchens-100.
Problem
Action anticipation methods ignore whether predicted future action sequences are plausible (temporally consistent and verb-noun valid) and tend to repeat actions over long horizons.
Approach
Generate future action sequences with a large video-language model (Q-Former + LLM), and train it with a contrastive-style loss against counterfactual sequences built from temporal logic and verb-noun pair constraints, plus a position-weighted penalty (increasing with token index) that discourages repetition over long horizons.
Key results
- On Ego4D long-term anticipation, PlausiVL reaches ED@Z=20 of 0.679 verb and 0.681 noun, better than Video-LLaMA (0.703/0.721).
- On EPIC-Kitchens-100, PlausiVL achieves 55.62 verb, 54.23 noun, and 27.60 action class-mean Top-5 recall, versus 26.05 action for Video-LLaMA.
- Repetition score drops from 7.12 (Video-LLaMA) to 5.87 for PlausiVL, while BLEU rises from 37.89 to 45.54.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the plausible action sequence learning loss (Lplau) (ablation) | PlausiVL · Ego4D | ED@(Z=20) verb: 0.691 -> 0.679 | Helped |
| Add the plausible action sequence learning loss (Lplau) (ablation) | PlausiVL · EPIC-Kitchens-100 | Class-mean Top-5 recall (%) verb: 54.15 -> 55.62 | Helped |
| Add the long-horizon action repetition loss (Lrep) (ablation) | PlausiVL · Ego4D | ED@(Z=20) noun: 0.698 -> 0.683 | Helped |
| Add the long-horizon action repetition loss (Lrep) (ablation) | PlausiVL · EPIC-Kitchens-100 | Class-mean Top-5 recall (%) action: 26.67 -> 27.60 | Helped |
| Use PlausiVL instead of Video LLaMA | Video-language action anticipation · EPIC-Kitchens-100 | Class-mean Top-5 recall (%) verb: 52.90 -> 55.62 | Helped |
| Use PlausiVL instead of Video-LLaMA | Video-language action anticipation · Ego4D | BLEU score: 37.89 -> 45.54 | Helped |
| Use PlausiVL instead of Video-LLaMA | Video-language action anticipation · Ego4D | Repetition Score: 7.12 -> 5.87 | Helped |
Limitations
- Authors note the method is an initial step and that hallucination of implausible action sequences remains an issue.
- Plausibility is evaluated indirectly via BLEU, repetition score and edit distance rather than a direct plausibility measure.
Experimental setup
Datasets: Ego4D EPIC-Kitchens-100
Benchmarks: Ego4D EPIC-Kitchens-100
Metrics: ED@Z=20 class-mean Top-5 recall BLEU repetition score
Baselines: Video-LLaMA Video-LLM AntGPT RepLAI SlowFast ICVAE HierVL RU-LSTM AVT MeMViT RAFTformer InAViT
Builds on: BLIP-2 DTL Video-LLaMA Video-LLM
Closest papers by meaning
- AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos? (ICLR 2024 · similarity 0.89)
- Uncertainty-aware Action Decoupling Transformer for Action Anticipation (CVPR 2024 · similarity 0.88)
- Towards Understanding Future: Consistency Guided Probabilistic Modeling for Action Anticipation (AAAI 2024 · similarity 0.87)
- RoboDreamer: Learning Compositional World Models for Robot Imagination (ICML 2024 · similarity 0.87)
- Latent Action Reparameterization for Efficient Agent Inference (NeurIPS 2026 · similarity 0.87)
- PlanLLM: Video Procedure Planning with Refinable Large Language Models (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.