Research map / Video / Action recognition
Language Model Guided Interpretable Video Action Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LaIAR, which uses a scene-graph-based language model branch with DT-Former to guide a RGB-only video model during training via joint embedding, token selection supervision, and cross-modal distillation, yielding an interpretable video action recognizer.
Problem
Interpretable-by-design video action recognition models underperform black-box models, and two-stage scene-graph-based reasoning pipelines optimize components separately.
Approach
Use a language (semantic scene graph) model branch with the same DT-Former architecture to guide the video model during training through visual-semantic joint embedding, token selection supervision, and KL-based cross-modal distillation, so that inference uses only RGB input while still providing nearest-neighbor semantic explanations.
Key results
- On Charades, LaIAR reaches 63.6% mAP with ground-truth relations versus 61.1% for the video-only DT-Former baseline, and 45.1 mAP in the RGB-only setting, beating OR2G (44.9).
- On CAD-120, LaIAR achieves 0.85 mAR, outperforming VideoMLN (0.83) and Explainable AAR-RAR (0.80).
- Under a domain-shift split of Charades, LaIAR averages 57.2% accuracy with variance 0.11, versus STIGPN's 54.1 with variance 0.30.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add visual-semantic joint embedding loss (Lsim) (ablation) | LaIAR · Charades | mAP: 61.1 -> 62.2 | Helped |
| add visual-semantic joint embedding loss (Lsim) (ablation) | LaIAR · CAD-120 | mAR: 0.75 -> 0.79 | Helped |
| add token selection supervision loss (Ltss) (ablation) | LaIAR · Charades | mAP: 61.1 -> 61.7 | Helped |
| add token selection supervision loss (Ltss) (ablation) | LaIAR · CAD-120 | mAR: 0.75 -> 0.79 | Helped |
| add cross-modal learning loss (Lxm) (ablation) | LaIAR · Charades | mAP: 61.1 -> 62.9 | Helped |
| add cross-modal learning loss (Lxm) (ablation) | LaIAR · CAD-120 | mAR: 0.75 -> 0.81 | Helped |
| add all learning-scheme components (Lsim, Ltss, and Lxm) (ablation) | LaIAR · Charades | mAP: 61.1 -> 63.6 | Helped |
| add all learning-scheme components (Lsim, Ltss, and Lxm) (ablation) | LaIAR · CAD-120 | mAR: 0.75 -> 0.85 | Helped |
| add spatial token selection (ablation) | DT-Former · Charades | mAP: 61.2 -> 61.1 | No clear effect |
| add temporal token selection (ablation) | DT-Former · Charades | mAP: 61.2 -> 61.1 | No clear effect |
| add spatial and temporal token selection (ablation) | DT-Former · CAD-120 | mAR: 0.73 -> 0.75 | Helped |
| use predicted rather than annotated semantic relationships (ablation) | LaIAR · Charades | mAP: 63.6 -> 62.4 | Hurt |
Surprising findings
- Using predicted (rather than ground-truth) semantic relationships during training improved the video model to 62.4 mAP versus 61.1 without the language branch.
Limitations
- Relies on scene graph annotations or a fine-tuned visual relationship detector for the semantic branch.
- Explanations rely on nearest-neighbor semantic labels in the joint embedding space, which is an indirect form of interpretability.
Experimental setup
Datasets: Charades Action Genome CAD-120
Benchmarks: Charades CAD-120
Metrics: mAP mAR
Baselines: I3D SlowFast LFB STRG SGFB VideoMLN OR2G Explainable AAR-RAR Temporal Segment
Builds on: Action Genome scene graphs ViT Gumbel-Softmax token selection Generalized Pooling Operator
Closest papers by meaning
- Look, Remember and Reason: Grounded Reasoning in Videos with Language Models (ICLR 2024 · similarity 0.87)
- LLaVAction: evaluating and training multi-modal large language models for action understanding (ICLR 2026 · similarity 0.86)
- Storyboard-guided Alignment for Fine-grained Video Action Recognition (NeurIPS 2025 · similarity 0.86)
- Disentangled Concepts Speak Louder Than Words: Explainable Video Action Recognition (NeurIPS 2025 · similarity 0.85)
- Efficient Transfer Learning for Video-language Foundation Models (CVPR 2025 · similarity 0.85)
- Kronecker Mask and Interpretive Prompts are Language-Action Video Learners (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.