Research map / Vision-language and multimodal / CLIP and zero-shot
SKI Models: Skeleton Induced Vision-Language Embeddings for Understanding Activities of Daily Living
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Integrates 3D skeleton features into CLIP-style video-language embeddings via SkeletonCLIP online distillation (SKI-VLM) and a skeleton projector for LVLMs (SKI-LVLM), enabling zero-shot ADL action recognition and dense captioning without skeletons at inference.
Problem
Video-language models trained on web videos generalize poorly to Activities of Daily Living (ADL) videos, which have similar appearances, subtle motion, and multiple viewpoints; prior skeleton-RGB methods are not zero-shot and lack language grounding.
Approach
Build a skeleton-language model (SkeletonCLIP, Hyperformer skeleton encoder + frozen CLIP text encoder) and distill it online into a VideoCLIP model within a language-contextualized space (SkeletonCLIP Distillation), so skeleton knowledge is available at inference without skeleton input; for LVLMs, add a trainable skeleton projector alongside the visual projector.
Key results
- SKI-ViFiCLIP reaches 52.0% on NTU60 48/12 and 77.5% on NTU120 110/10 zero-shot, outperforming ViFiCLIP (48.2, 70.0) and other baselines.
- SKI-VLM improves over its baselines by up to +14.3% (XCLIP), +10.7% (ViFiCLIP), and +5.1% (FROSTER) on zero-shot recognition.
- SKI-LVLM averages 43.5 on Charades dense captioning versus 30.7 for VideoChatGPT and 25.4 for NTU-trained VideoChatGPT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply SkeletonCLIP Distillation to ViFiCLIP | ViFiCLIP · NTU60 48/12 | accuracy: 48.2 -> 52.0 | Helped |
| Apply SkeletonCLIP Distillation to ViFiCLIP | ViFiCLIP · NTU120 110/10 | accuracy: 70.0 -> 77.5 | Helped |
| Add skeleton features during LVLM training | VideoChatGPT on Charades · Charades | Average: 25.4 -> 43.5 | Helped |
| Use raw skeleton features instead of language-contextualized skeleton features (ablation) | VideoChatGPT on Charades · Charades | Average: 43.4 -> 43.5 | No clear effect |
| Use online rather than offline distillation (ablation) | SKI-ViFiCLIP · NTU120 110/10 | accuracy: 70.8 -> 77.5 | Helped |
| Use online rather than feature-level distillation without projection (ablation) | SKI-ViFiCLIP · NTU120 110/10 | accuracy: 65.3 -> 77.5 | Helped |
| Add a projection layer to feature-level distillation (ablation) | SKI-ViFiCLIP · NTU60 48/12 | accuracy: 52.9 -> 53.2 | Helped |
| Replace contrastive distillation loss with MSE (ablation) | SKI-ViFiCLIP · NTU120 110/10 | accuracy: 70.0 -> 77.5 | Helped |
| Replace KL divergence distillation loss with MSE (ablation) | SKI-ViFiCLIP · NTU120 110/10 | accuracy: 73.1 -> 77.5 | Helped |
| Freeze SkeletonCLIP text encoder rather than train it (ablation) | SKI-ViFiCLIP · Harmonic Mean | accuracy: 60.7 -> 62.2 | Helped |
| Pretrain both SkeletonCLIP and ViFiCLIP rather than neither (ablation) | SKI-ViFiCLIP · Harmonic Mean | accuracy: 16.7 -> 62.2 | Helped |
| Add SkeletonCLIP pretraining while ViFiCLIP remains pretrained (ablation) | SKI-ViFiCLIP · Harmonic Mean | accuracy: 60.2 -> 62.2 | Helped |
Surprising findings
- Language-contextualized skeleton features gave no clear advantage in the LVLM setting: SK-VCGPT (raw skeleton features) performed comparably to SKI-LVLM (43.4 vs 43.5 average).
- Naively fusing SkeletonCLIP features directly with VLM features reduced accuracy (17.5 on NTU60 48/12 for ViFiCLIP + SkeletonCLIP), attributed to conflicting gradients.
Limitations
- Skeleton data is required during training, and the approach relies on synchronized skeleton-video-text data that is scarce.
- Zero-shot results on NTU60/NTU120 are evaluated on unseen-class splits of a lab-collected dataset, and LVLM evaluation uses LLM-judged scores on Charades rather than human judgments.
- Skeleton encoder choice depends on pretraining; ablations show performance drops without skeleton pretraining (e.g., 10.0 on NTU120 without pretraining).
Experimental setup
Datasets: NTU RGB+D 60 NTU RGB+D 120 Charades ActivityNet
Benchmarks: NTU RGB+D 60 NTU RGB+D 120 Charades
Metrics: zero-shot top-1 accuracy VideoChatGPT five metrics (correctness, detail orientation, harmonic mean
Baselines: SynSE SMIE STAR XCLIP FROSTER ViFiCLIP VideoChatGPT Feature-level KD Offline KD
Builds on: CLIP ViFiCLIP XCLIP FROSTER Hyperformer VideoChatGPT
Closest papers by meaning
- LLAVIDAL: A Large LAnguage VIsion Model for Daily Activities of Living (CVPR 2025 · similarity 0.88)
- Part-aware Unified Representation of Language and Skeleton for Zero-shot Action Recognition (CVPR 2024 · similarity 0.87)
- Boosting Skeleton-based Zero-Shot Action Recognition with Training-Free Test-Time Adaptation (NeurIPS 2025 · similarity 0.87)
- Universal Skeleton Understanding via Differentiable Rendering and MLLMs (ICML 2026 · similarity 0.86)
- Neuron: Learning Context-Aware Evolving Representations for Zero-Shot Skeleton Action Recognition (CVPR 2025 · similarity 0.86)
- Heterogeneous Skeleton-Based Action Representation Learning (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.