Research map / Vision-language and multimodal / Multimodal LLMs

LLaVAction: evaluating and training multi-modal large language models for action understanding

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds EPIC-KITCHENS-100-MQA, a hard-distractor multiple-choice benchmark mined with a specialist action recognizer (TIM), and trains LLaVAction (LLaVA-Video/OV based) with a GPT-4o-reformulated action dataset, an action token, and a two-stage structured action pipeline.

Problem

MLLMs' fine-grained action understanding is not well examined; free-text action outputs are hard to match to structured action label sets (~4k classes in EPIC-KITCHENS-100) and make comparison with specialist action recognizers difficult.

Approach

Use specialist action recognition models (TIM, AVION) to mine hard distractors for MQA evaluation and training, add a learnable action token supervised with verb/noun/action classification heads to boost visual attention, and use a two-stage top-K candidate pipeline to output structured actions.

Key results

• LLaVAction-7B (LLaVA-Video base) reaches 73.4% on EPIC-KITCHENS-100-MQA with 16 frames versus 52.2% for GPT-4o, a gain of about 21 points.
• On EPIC-KITCHENS-100 validation, LLaVAction-7B achieves 58.3% (action label) and 63.2% (action narration) top-1, exceeding TIM at 56.4%.
• LLaVAction-7B outperforms LLaVA-Video-7B on 10 of 13 MLLM benchmarks, e.g. CVRR 55.6 vs 51.3 and MVBench 61.1 vs 58.6.

Surprising findings

• GPT-4o drops from 87.6% with random distractors to 52.2% with TIM-mined hard distractors, showing that random-distractor MQA overstates MLLM action understanding.
• Naively fine-tuning LLaVA-Video-7B on GPT-4o-based reformulated data degraded EPIC-KITCHENS-100-MQA accuracy from 35.7% to 21.9%, suggesting catastrophic forgetting of MQA capability.

Limitations

• Two-stage pipeline upper-bounds performance by whether the ground-truth action is in the top-K candidates from the specialist model.
• Benchmark and training distractors depend on specialist models (TIM, AVION), so error distributions of those models shape the evaluation.
• Gains do not extend to all general video benchmarks; e.g., VideoMME and some sub-categories show declines.

Experimental setup

Datasets: EPIC-KITCHENS-100 LLaVA-Video-178K EPFL-Smart-Kitchen-30 MECCANO Animal Kingdom

Benchmarks: EPIC-KITCHENS-100-MQA EPIC-KITCHENS-100 EPFL-Smart-Kitchen-30 MECCANO Animal Kingdom VDC VideoDC VideoEval-Pro ActivityNet-QA VideoChatGPT CVRR TempCompass

Metrics: accuracy top-1 accuracy Jaccard accuracy

Baselines: GPT-4o GPT-4o-mini LLaVA-Video-7B LLaVA-OV-7B LLaVA-OV-0.5B AVION TIM LaViLa VideoMAE Multi-modal VideoMAE SlowFast InternVL3-8B

Builds on: LLaVA-Video LLaVA-OneVision TIM AVION LMMs-Eval

Closest papers by meaning

• MA-Bench: Towards Fine-grained Micro-Action Understanding (CVPR 2026 · similarity 0.90)
• Grounding Multimodal Large Language Models in Actions (NeurIPS 2024 · similarity 0.89)
• On Discriminative vs. Generative classifiers: Rethinking MLLMs for Action Understanding (ICLR 2026 · similarity 0.88)
• Weakly Supervised Temporal Action Localization via Dual-Prior Collaborative Learning Guided by Multimodal Large Language Models (CVPR 2025 · similarity 0.88)
• SeFAR: Semi-supervised Fine-grained Action Recognition with Temporal Perturbation and Learning Stabilization (AAAI 2025 · similarity 0.88)
• ActionAtlas: A VideoQA Benchmark for Domain-specialized Action Recognition (NeurIPS 2024 · similarity 0.87)