Research map / Vision-language and multimodal / Human and embodied modeling
Narrative Action Evaluation with Prompt-Guided Multimodal Interaction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces narrative action evaluation (NAE), generating detailed professional commentary on action execution, with a prompt-guided multimodal interaction framework and re-annotated MTL-AQA and FineGym datasets.
Problem
Generating narrative commentary that both describes an action in rich natural language and provides rigorous evaluation (scores, action types, qualitative judgments) is hard, since evaluation information occupies a small share of the text and multi-task learning yields little gain.
Approach
Reformulates score prediction as video-text matching by turning score intervals into learnable text prompts, uses two cross-attention transformers for bidirectional video-prompt interaction, and feeds predicted score/action tokens as a learnable template into a text decoder with a Tri-Token attention mask.
Key results
- On MTL-NAE the proposed method reaches NAE mAP 0.383 vs 0.261 for SwinBert (about 46.7% relative improvement).
- On FineGym-NAE NAE mAP improves from 0.118 (SwinBert) to 0.162, a 37.3% gain.
- Ablation: removing the Score-Aware Transformer drops mAP from 0.383 to 0.347; removing Context-Aware Transformer drops it to 0.352.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Prompt-Guided Multimodal Interaction instead of SwinBert | NAE model · MTL-NAE | mAP: 0.261 -> 0.383 | Helped |
| use Prompt-Guided Multimodal Interaction instead of SwinBert | NAE model · FineGym-NAE | mAP: 0.118 -> 0.162 | Helped |
| add Context-Aware Transformer (CAT) (ablation) | Ours · MTL-NAE | mAP: 0.352 -> 0.383 | Helped |
| add Score-Aware Transformer (SAT) (ablation) | Ours · MTL-NAE | mAP: 0.347 -> 0.383 | Helped |
| add cross-entropy loss (CE) (ablation) | Ours · MTL-NAE | mAP: 0.308 -> 0.383 | Helped |
| add mean squared error loss (MSE) (ablation) | Ours · MTL-NAE | mAP: 0.316 -> 0.383 | Helped |
| add both CE and MSE losses (ablation) | Ours · MTL-NAE | mAP: 0.293 -> 0.383 | Helped |
| add learnable tokens to the template containing fixed evaluation tokens (ablation) | NAE text generator · MTL-NAE | mAP: 0.364 -> 0.383 | Helped |
| use a mixed evaluation-and-learnable template instead of an all-learnable template (ablation) | NAE text generator · MTL-NAE | mAP: 0.258 -> 0.383 | Helped |
| add AQA and action classification tasks to captioning-only training (ablation) | multi-task model · MTL-NAE | mAP: 0.341 -> 0.383 | Helped |
Surprising findings
- Naive multi-task learning (as in MTL-AQA baselines) improves single tasks very little and sometimes degrades them.
Limitations
- Re-annotated text labels are generated by ChatGPT and checked by annotators, so they depend on LLM generation quality.
- Multi-video AQA methods outperform the single-video approach on score prediction, and the approach is limited to single-video input.
Experimental setup
Datasets: MTL-AQA FineGym
Benchmarks: MTL-NAE FineGym-NAE
Metrics: mAP Spearman's rho R-l2 BLEU-4 METEOR CIDEr action accuracy
Baselines: C3D-AVG MSCADC UniVL VLCap VLTinT SwinBert USDL MUSDL CoRe TPT
Builds on: SwinBERT CLIP Video Swin Transformer MTL-AQA multi-task AQA
Closest papers by meaning
- NarrativeBridge: Enhancing Video Captioning with Causal-Temporal Narrative (ICLR 2025 · similarity 0.84)
- MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks (ICLR 2026 · similarity 0.84)
- ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate (ICLR 2024 · similarity 0.83)
- WorldMemArena: Evaluating Multimodal Agent Memory Through Action–World Interaction (NeurIPS 2026 · similarity 0.83)
- MULTISCRIPT: Multimodal Script Learning for Supporting Open Domain Everyday Tasks (AAAI 2024 · similarity 0.83)
- LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.