Research map / Vision-language and multimodal / Multimodal LLMs
Universal Skeleton Understanding via Differentiable Rendering and MLLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SkeletonLLM, which renders skeleton sequences into images via a learnable differentiable 3DGS+LBS renderer (DrAction) and trains InternVL3-8B with a four-stage cooperative curriculum (MQA warm-up, Disc-FT, CR-Distill, refinement) for format-agnostic skeleton action recognition and captioning.
Problem
MLLMs cannot natively process skeleton sequences; existing feature-alignment and tokenization methods are lossy and tied to specific skeleton formats, limiting open-vocabulary recognition, cross-format transfer, and semantic understanding.
Approach
Translate skeleton sequences of any format into images with a differentiable renderer built on 3D Gaussian Splatting and linear blend skinning, so MLLM task gradients can optimize the rendering into task-informative visual tokens; combined with a staged training curriculum of discriminative binary QA and causal-reasoning distillation.
Key results
- On NTU-60 30/30 split, SkeletonLLM reaches 37.84% vs 25.88% for TDSM (+11.96%), and 87.37% on the 55/5 split.
- Cross-format transfer NTU-60 to NW-UCLA reaches 60.38% accuracy vs 43.19% for TDSM.
- Removing CR-Distill drops 30/30 accuracy from 37.84% to 35.90%; joint training from scratch drops it to 34.10%.
Surprising findings
- Fixed JTM rendering performs much worse than simpler 3D+Velocity rendering, as the MLLM struggles with dense visual patterns.
- Rendering skeletons with DrAction and feeding them to GPT-5.4 without joint training (53.18% on 48/12) still lags the jointly trained SkeletonLLM.
Limitations
- Progressive four-stage training is needed because joint end-to-end training is unstable.
- Cross-format transfer is evaluated on a limited set of source-target pairs.
- Current implementation is an engineering version without inference-specific optimizations.
Experimental setup
Datasets: NTU-60 NTU-120 PKU-MMD NW-UCLA HumanML3D NTU-60 (2D)
Benchmarks: NTU-60 NTU-120 PKU-MMD NW-UCLA HumanML3D Skeleton-QA
Metrics: top-1 accuracy R@1 R@3 MM-Dist BLEU@4 ROUGE-L CIDEr BertScore
Baselines: ReViSE JPoSE CADA-VAE SynSE PURLS SCoPLe TDSM MotionGPT MotionLLM Qwen2.5-VL-7B InternVL3-8B SKI-LVLM
Builds on: 3D Gaussian Splatting Linear Blend Skinning InternVL3 MotionGPT MotionLLM
Closest papers by meaning
- SAMoR: Motion Modelling for Articulated Objects of Any Skeleton and Topology (NeurIPS 2026 · similarity 0.88)
- Part-aware Unified Representation of Language and Skeleton for Zero-shot Action Recognition (CVPR 2024 · similarity 0.87)
- Heterogeneous Skeleton-Based Action Representation Learning (CVPR 2025 · similarity 0.87)
- SKI Models: Skeleton Induced Vision-Language Embeddings for Understanding Activities of Daily Living (AAAI 2025 · similarity 0.86)
- JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images (NeurIPS 2024 · similarity 0.86)
- Less is More: Compact-Token Masked Feature Learning for Skeleton Representation Learning (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.