Research map / Video / Action recognition
PALM: Predicting Actions through Language Models
Sanghwan Kim, Daoji Huang, Yongqin Xian, Otmar Hilliges, Luc Van Gool, Xi Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MMR exemplar selection instead of random selection (ablation) | PALM · Ego4D v2 val | Action ED: 0.9002 -> 0.8908 | Improved |
| use MMR instead of similarity-only selection (ablation) | PALM · Ego4D v2 val | Action ED: 0.8923 -> 0.8908 | Improved |
| add image captions to the action-sequence prompt (ablation) | PALM · Ego4D v2 val | Action ED: 0.8947 -> 0.8908 | Improved |
| add past action sequence to the captions-only prompt (ablation) | PALM · Ego4D v2 val | Action ED: 0.9064 -> 0.8908 | Improved |
| use a larger LLM (Llama 2 13B instead of 7B) (ablation) | PALM · Ego4D v2 val | Action ED: 0.8908 -> 0.8867 | Improved |
| use EgoVLP video encoder instead of SlowFast for action recognition (ablation) | PALM · Ego4D v2 val | Action ED: 0.9278 -> 0.8819 | Improved |
| use ground truth past actions instead of recognized actions (ablation) | PALM · Ego4D v2 val | Action ED: 0.8819 -> 0.8319 | Improved |
| increase in-context exemplars from 1 to 4 (ablation) | PALM · Ego4D v2 val | Action ED: 0.8908 -> 0.8859 | Improved |
| use 4 input segments instead of 1 for action recognition (ablation) | action recognition module · Ego4D v2 val | action accuracy: 17.02 -> 20.63 | Improved |
| use 8 input segments instead of 4 (ablation) | action recognition module · Ego4D v2 val | action accuracy: 20.63 -> 17.57 | Hurt |
| use classification heads instead of video-text retrieval for action recognition (ablation) | action recognition module · Ego4D v2 val | action accuracy: 15.18 -> 20.63 | Improved |
| swap the captioning model (BLIP to VideoBLIP) (ablation) | PALM · Ego4D v2 val | Action ED: 0.8927 -> 0.8907 | No clear effect |
Later papers that compared with this method
4 later accepted papers report PALM as a baseline in their results tables; 3 of them report a better number on most of the results they share with it.
- Open-Vocabulary Long Term Action Anticipation (ECCV 2026 · better on 0 of 2 shared results)
- AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos? (ICLR 2024 · better on 2 of 2 shared results)
- Vamos: Versatile Action Models for Video Understanding (ECCV 2024 · better on 1 of 1 shared result)
- Text-Conditioned Resampler For Long Form Video Understanding (ECCV 2024 · better on 1 of 1 shared result)
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- AntGPT — AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos? (ICLR 2024) · better on 4 of 5 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Open-Vocabulary Long Term Action Anticipation (ECCV 2026 · similarity 0.90)
- About Time: Advances, Challenges, and Outlooks of Action Understanding (IJCV 2025 · similarity 0.89)
- Summarize the Past to Predict the Future: Natural Language Descriptions of Context Boost Multimodal Object Interaction Anticipation (CVPR 2024 · similarity 0.87)
- AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos? (ICLR 2024 · similarity 0.87)
- LLaVAction: evaluating and training multi-modal large language models for action understanding (ICLR 2026 · similarity 0.87)
- Do Egocentric Video-Language Models Truly Understand Hand-Object Interactions? (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
