Research map / Vision-language and multimodal / Multimodal LLMs
HIS-GPT: Towards 3D Human-In-Scene Multimodal Understanding
Jiahe Zhao, Ruibing Hou, Zejie Tian, Hong Chang, Shiguang Shan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add AInt auxiliary tasks (act, spa, cont) (ablation) | HIS-GPT with sine PE · HIS-Bench (avg) | score (0-100): 43.0 -> 44.1 | Improved |
| add LTP position encoding (ablation) | HIS-GPT · HIS-Bench (avg) | score (0-100): 43.0 -> 46.0 | Improved |
| add both AInt and LTP (ablation) | HIS-GPT · HIS-Bench (avg) | score (0-100): 43.0 -> 48.7 | Improved |
| add activity classification task only (ablation) | HIS-GPT with LTP · HIS-Bench (avg) | score (0-100): 46.0 -> 45.3 | Hurt |
| add spatial relation task only (ablation) | HIS-GPT with LTP · HIS-Bench (avg) | score (0-100): 46.0 -> 47.3 | Improved |
| add contact detection task only (ablation) | HIS-GPT with LTP · HIS-Bench (avg) | score (0-100): 46.0 -> 46.9 | Improved |
| add stage 2 instruction tuning (ablation) | HIS-GPT · HIS-Bench (avg) | score (0-100): 37.5 -> 48.7 | Improved |
| add stage 1 modality alignment (ablation) | HIS-GPT · HIS-Bench (avg) | score (0-100): 42.6 -> 48.7 | Improved |
| add scene and motion caption data in stage 1 (ablation) | HIS-GPT · HIS-Bench (avg) | score (0-100): 45.8 -> 48.7 | Improved |
| use LoRA for LLM in stage 2 (ablation) | HIS-GPT · HIS-Bench (avg) | score (0-100): 48.7 -> 46.6 | Hurt |
| use LoRA for LLM in both stages (ablation) | HIS-GPT · HIS-Bench (avg) | score (0-100): 48.7 -> 38.1 | Hurt |
| render 3D data to video instead of RGB video (ablation) | GPT-4o · HIS-Bench (avg) | score (0-100): 28.3 -> 31.3 | Improved |
Closest papers by meaning
- Multi-modal Situated Reasoning in 3D Scenes (NeurIPS 2024 · similarity 0.91)
- HSI-GPT: A General-Purpose Large Scene-Motion-Language Model for Human Scene Interaction (CVPR 2025 · similarity 0.89)
- Hypo3D: Exploring Hypothetical Reasoning in 3D (ICML 2025 · similarity 0.89)
- Situational Awareness Matters in 3D Vision Language Reasoning (CVPR 2024 · similarity 0.89)
- Embodied Scene Understanding for Vision Language Models via MetaVQA (CVPR 2025 · similarity 0.89)
- EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
