Research map / Vision-language and multimodal / Multimodal LLMs
HoloLLM: Multisensory Foundation Model for Language-Grounded Human Sensing and Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HoloLLM, an MLLM that adds LiDAR, infrared, mmWave, WiFi and RFID sensing to an LLM via a Universal Modality-Injection Projector that fuses CLIP-based initial embeddings with tailored-encoder features, evaluated on new language-grounded benchmarks built from MM-Fi and XRF55.
Problem
Multimodal LLMs for human sensing lack support for rare, data-scarce sensing modalities (LiDAR, mmWave, WiFi, RFID, infrared) whose heterogeneous signals are hard to align with text given few paired samples, and no language-grounded sensing benchmark exists.
Approach
Use a frozen-style CLIP encoder to produce text-pre-aligned initial embeddings per modality, then iteratively inject fine-grained features from modality-specific tailored encoders into these embeddings via coarse-to-fine cross-attention in a projector (UMIP), avoiding large modality-text pretraining.
Key results
- On MM-Fi CrossEnv, HoloLLM achieves 56.4% average Action QA accuracy vs. 16.7% for ImageBind and 5.0% for OneLLM.
- On MM-Fi Random, HoloLLM reaches 86.5% average Action QA accuracy and 28.4 METEOR captioning, outperforming all baselines.
- Ablation on MM-Fi CrossEnv: tailored encoders raise average QA accuracy from 6.2% to 46.6%, and UMIP further raises it to 56.4%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace CLIP ViT-L with tailored encoders (ablation) | HoloLLM baseline · MM-Fi CrossEnv | Action QA accuracy: 6.2 -> 46.6 | Helped |
| replace CLIP ViT-L with tailored encoders (ablation) | HoloLLM baseline · XRF55 CrossEnv | Action QA accuracy: 3.5 -> 11.8 | Helped |
| add UMIP to tailored encoders (ablation) | HoloLLM · MM-Fi CrossEnv | Action QA accuracy: 46.6 -> 56.4 | Helped |
| add UMIP to tailored encoders (ablation) | HoloLLM · XRF55 CrossEnv | Action QA accuracy: 11.8 -> 12.4 | Helped |
| add UMIP to tailored encoders (ablation) | HoloLLM · MM-Fi CrossEnv | Action Recognition accuracy: 58.5 -> 59.5 | Helped |
| add UMIP to tailored encoders (ablation) | HoloLLM · XRF55 CrossEnv | Action Caption METEOR: 14.9 -> 15.2 | Helped |
| use HoloLLM rather than ImageBind | multisensory MLLMs · MM-Fi Random | Action QA accuracy: 46.2 -> 86.5 | Helped |
| use HoloLLM rather than ImageBind | multisensory MLLMs · XRF55 Random | Action QA accuracy: 35.8 -> 63.5 | Helped |
| naively concatenate multimodal tokens | HoloLLM, MM-Fi CrossEnv Vision and mmWave · MM-Fi CrossEnv | Action Recognition accuracy: 80.6 -> 84.6 | Helped |
| naively concatenate multimodal tokens | HoloLLM, MM-Fi CrossEnv Vision and mmWave · MM-Fi CrossEnv | Action QA accuracy: 79.5 -> 66.4 | Hurt |
| use weighted-sum fusion instead of single Vision modality | HoloLLM, MM-Fi CrossEnv Vision and mmWave · MM-Fi CrossEnv | Action QA accuracy: 79.5 -> 81.4 | Helped |
| use weighted-sum fusion instead of single Vision modality | HoloLLM, XRF55 CrossEnv Vision and Infrared · XRF55 CrossEnv | Action Caption METEOR: 19.8 -> 20.6 | Helped |
Surprising findings
- Naive multimodal fusion of tokens improves action recognition for some modality pairs but fails to improve action QA and captioning.
- ImageBind, an encoder-based method, outperforms projector-based MLLMs such as OneLLM and Honeybee by a wide margin on several sensing modalities.
Limitations
- Scope limited to action recognition, action QA and captioning; no task planning or action generation.
- Cross-subject and cross-environment generalization remains weak for some modalities such as WiFi (e.g., 8-11% QA accuracy).
- Captions are generated automatically by LLaVA-Video from video, so sensing modalities are annotated indirectly via the vision stream.
Experimental setup
Datasets: MM-Fi XRF55
Benchmarks: MM-Fi (Random, CrossSub, CrossEnv) XRF55 (Random, CrossSub, CrossEnv)
Metrics: accuracy METEOR
Baselines: Tokenpacker Honeybee OneLLM ImageBind
Builds on: OneLLM Q-Former CLIP LLaVA-Video
Closest papers by meaning
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.86)
- HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses Through Reasoning MLLMs (AAAI 2026 · similarity 0.86)
- SensorLM: Learning the Language of Wearable Sensors (NeurIPS 2025 · similarity 0.85)
- MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World (CVPR 2024 · similarity 0.85)
- A Touch, Vision, and Language Dataset for Multimodal Alignment (ICML 2024 · similarity 0.85)
- UP-VLA: A Unified Understanding and Prediction Model for Embodied Agent (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.