Research map / Vision-language and multimodal / Multimodal LLMs
LLAVIDAL: A Large LAnguage VIsion Model for Daily Activities of Living
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Curates ADL-X, a 100K multiview RGB+skeleton instruction-tuning dataset from NTU RGB+D 120, and trains LLAVIDAL, an LLVM that adds skeleton and object (HOI) features via a progressive modality-integration (MMPro) curriculum.
Problem
Web-video-trained LLVMs struggle with fine-grained details, complex human-object interactions, and view-invariant understanding needed for Activities of Daily Living (ADL) videos, due to lack of ADL instruction data and insufficient modality integration.
Approach
A semi-automated ADL instruction-data pipeline (person-augmented cropping, temporal stitching of trimmed clips, weakly supervised descriptions) plus a three-stage curriculum (video, then skeleton, then object) that aligns skeleton and HOI features into the LLM embedding space via modality-specific connectors.
Key results
- LLAVIDAL reaches 55.2 on Charades-based ADL MCQ-AR (Char) and 48.1 on Smarthome, vs 51.0 and 39.6 for VideoChatGPT.
- On TSU description average, LLAVIDAL scores 70.8 vs 54.5 for VideoChatGPT and 61.9 for ChatUniVi.
- Progressive MMPro training (token prefixing) beats joint X-InstructBLIP-style integration, e.g. 38.2 vs 29.9 on TSU MCQ-TC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on ADL-X rather than ActivityNet | LLVM · Toyota Smarthome Untrimmed video description | average score: 31.9 -> 64.8 | Helped |
| Add person-centric cropping (PAG) to ADL-X curation (ablation) | ADL-X-trained LLVM · Toyota Smarthome Untrimmed video description | average score: 59.1 -> 64.8 | Helped |
| Add weak supervision (WS) to ADL-X curation (ablation) | ADL-X-trained LLVM · Toyota Smarthome Untrimmed video description | average score: 62.2 -> 64.8 | Helped |
| Add skeleton features (SF) (ablation) | LLAVIDAL · ADL MCQ-TC LEMMA | accuracy: 28.6 -> 33.1 | Helped |
| Add skeleton context and features (SC + SF) (ablation) | LLAVIDAL · ADL MCQ-AR Toyota Smarthome | accuracy: 44.5 -> 46.2 | Helped |
| Add HOI features (ablation) | LLAVIDAL · ADL MCQ-TC Toyota Smarthome Untrimmed | accuracy: 29.5 -> 37.1 | Helped |
| Add skeleton or HOI as QA or context rather than features (ablation) | LLAVIDAL · ADL MCQ-TC Toyota Smarthome Untrimmed | accuracy: 29.5 -> 28.4 | Hurt |
| Use MMPro progressive integration rather than joint multimodal alignment | LLAVIDAL · Toyota Smarthome Untrimmed video description | average score: 65.5 -> 70.8 | Helped |
| Integrate skeleton before HOI rather than HOI before skeleton (ablation) | MMPro · ADL MCQ-TC Toyota Smarthome Untrimmed | accuracy: 33.0 -> 38.2 | Helped |
| Use modality token prefixes rather than string prefixes (ablation) | MMPro Prog. A · Toyota Smarthome Untrimmed video description | average score: 65.5 -> 70.8 | Helped |
| Add skeleton context alongside skeleton features (ablation) | MMPro Prog. A · ADL MCQ-TC Toyota Smarthome Untrimmed | accuracy: 38.2 -> 34.9 | Hurt |
| Add HOI tokens (ablation) | Video-only model · use laptop action recognition | improvement: +45% | Helped |
Surprising findings
- Skeleton context text adds little once skeleton features are used; skeleton features alone are sufficient.
- Joint alignment of all modalities underperforms the video+skeleton/HOI model trained on fewer modalities.
Limitations
- Training data derived from a single source (NTU RGB+D 120) with synthetic GPT-3.5 annotations, risking hallucination.
- Human quality check covered only 100 QA pairs.
Experimental setup
Datasets: NTU RGB+D 120 ActivityNet
Benchmarks: ADL MCQ-AR ADL MCQ-TC Charades TSU LEMMA
Metrics: accuracy LLM-judged description score (CI, DO, CU, TU, Consistency)
Baselines: VideoChatGPT VideoLLaVA VideoLLaMA ChatUniVi CogVLM + GPT BLIP2 + GPT X-InstructBLIP
Builds on: Video-ChatGPT X-InstructBLIP CogVLM SkeletonCLIP OWLv2
Closest papers by meaning
- LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning (CVPR 2026 · similarity 0.88)
- Understanding Long Videos with Multimodal Language Models (ICLR 2025 · similarity 0.88)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.88)
- SKI Models: Skeleton Induced Vision-Language Embeddings for Understanding Activities of Daily Living (AAAI 2025 · similarity 0.88)
- LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token (ICLR 2025 · similarity 0.88)
- Unleashing Hour-Scale Video Training for Long Video-Language Understanding (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.