Research map / Large language models / Language, safety and interpretability
LLMs are Good Action Recognizers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Treats a frozen LLaMA-13B as a skeleton action recognizer by projecting skeleton sequences into discrete "action sentences" via a hyperbolic-codebook VQ-VAE with human-inductive-bias losses, then fine-tuning only LoRA adapters.
Problem
Skeleton-based action recognition needs accurate recognizers; the paper asks whether a large language model can serve as the recognizer if skeleton inputs are converted into a sentence-like format, while keeping the LLM's pre-trained weights intact.
Approach
A linguistic projection maps skeleton sequences to discrete token sequences ("action sentences") using an action-based VQ-VAE with a Poincaré-ball hyperbolic codebook, regularized by Zipf's law, context-sensitivity (paired-token) losses, and MMD alignment to LLM word embeddings; the LLM is then adapted only with LoRA.
Key results
- Achieves 95.0% X-Sub and 98.4% X-View on NTU RGB+D, and 88.7% X-Sub and 91.5% X-Set on NTU RGB+D 120, beating SkeleTR (94.8/97.7/87.8/88.3).
- Reaches 67.0/36.1/66.6 on Toyota Smarthome and 46.3 on UAV-Human X-Sub, both best among compared methods.
- Full fine-tuning of the LLM drops accuracy to 79.6 vs 91.5 for LoRA-based LLM-AR on NTU-120 X-Set ablation; removing discretization gives 83.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Zipf, context-sensitivity, and MMD alignment losses (ablation) | LLM-AR · NTU RGB+D 120 X-Set | accuracy: 87.6 -> 91.5 | Helped |
| add the Zipf loss (ablation) | LLM-AR · NTU RGB+D 120 X-Set | accuracy: 89.9 -> 91.5 | Helped |
| add the context-sensitivity loss (ablation) | LLM-AR · NTU RGB+D 120 X-Set | accuracy: 89.8 -> 91.5 | Helped |
| add MMD alignment (ablation) | LLM-AR · NTU RGB+D 120 X-Set | accuracy: 90.3 -> 91.5 | Helped |
| discretize encoded latent features into action sentences (ablation) | LLM-AR · NTU RGB+D 120 X-Set | accuracy: 83.4 -> 91.5 | Helped |
| use a hyperbolic codebook instead of a Euclidean codebook (ablation) | LLM-AR · NTU RGB+D 120 X-Set | accuracy: 89.7 -> 91.5 | Helped |
| use LoRA instead of fine-tuning all LLM parameters (ablation) | LLM-AR · NTU RGB+D 120 X-Set | accuracy: 79.6 -> 91.5 | Helped |
| use LoRA instead of fine-tuning all LLM parameters (ablation) | LLM-AR · NTU RGB+D 120 unseen classes | accuracy: 37.7 -> 62.4 | Helped |
Surprising findings
- Fully fine-tuning the LLM performs much worse (79.6%) than keeping weights frozen with LoRA (91.5%), suggesting pre-trained knowledge is valuable here.
- On unseen activity classes (3-way random splits of NTU-120), the method reaches 62.4% versus 37.7% for full tuning.
Limitations
- Evaluation limited to four skeleton datasets with skeleton-only input.
- Relies on a large 13B LLM and a VQ-VAE pipeline, adding substantial compute.
- Benefit of LLM knowledge is shown only through ablations; the mechanism is argued qualitatively.
Experimental setup
Datasets: NTU RGB+D NTU RGB+D 120 Toyota Smarthome UAV-Human
Benchmarks: NTU RGB+D NTU RGB+D 120 Toyota Smarthome UAV-Human
Metrics: top-1 accuracy
Baselines: ST-GCN Shift-GCN InfoGCN PoseC3D SkeleTR DSTformer UNIK 2s-AGCN
Builds on: VQ-VAE LoRA Poincaré embeddings LLaMA
Closest papers by meaning
- LLaVAction: evaluating and training multi-modal large language models for action understanding (ICLR 2026 · similarity 0.87)
- LLM+AL: Bridging Large Language Models and Action Languages for Complex Reasoning About Actions (AAAI 2025 · similarity 0.87)
- Understanding Emotional Body Expressions via Large Language Models (AAAI 2025 · similarity 0.86)
- Grounding Multimodal Large Language Models in Actions (NeurIPS 2024 · similarity 0.85)
- Speech Recognition Meets Large Language Model: Benchmarking, Models, and Exploration (AAAI 2025 · similarity 0.85)
- Part-aware Unified Representation of Language and Skeleton for Zero-shot Action Recognition (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.