Research map / Vision-language and multimodal / Multimodal LLMs

HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses Through Reasoning MLLMs

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces HumanSense, a 15-task benchmark (3,882 questions) for human-centered perception and empathetic response in MLLMs, and a multi-stage omni-modal GRPO model (HumanSense-Omni-Reasoning) built on Qwen2.5-Omni-7B that improves high-level understanding and interaction tasks.

Problem

Existing MLLM benchmarks rarely evaluate human-centered perception, implicit intent understanding, and empathetic, context-aware responses in multi-turn interactions, leaving fine-grained evaluation of these abilities lacking.

Approach

A hierarchical four-tier benchmark (perception, complex perception, contextual understanding, feedback strategy) plus a modality-progressive GRPO training (video, then audio, then video+audio) that teaches reasoning before responding; a reasoning-style prompt is also shown to help non-reasoning models training-free.

Key results

• Human evaluators reach 87.5% average accuracy on HumanSense (tiny), about 29.7 points above the best model.
• Multi-stage omni-modal GRPO on Qwen2.5-Omni-7B raises the average from 0.559 to 0.619 across tasks (Avg.*, all tasks), with L2 average rising from 0.567 to 0.603 and PC from 0.399 to 0.619.
• Prompt enhancement of non-reasoning models improves high-level tasks training-free, e.g. Psychological Chat for Qwen2.5-Omni rises from 0.399 to 0.436 (FR2 from 0.740 to 0.763).

Surprising findings

• Visual-only models all score roughly 40-60 on high-level L3-L4 tasks with little variation, suggesting visual input alone is insufficient.
• Audio-only Qwen2-Audio depends heavily on ASR text, while the omni model gains little from text input.

Limitations

• Substantial gap remains between models and human performance, especially on L3-L4 tasks.
• Human-level comparison uses a small 450-question tiny subset (30 per task).
• Psychological Chat and emotional response tasks are multiple-choice, not open-ended generation.

Experimental setup

Datasets: CelebV-HQ AVA Jester ActivityNet UCF-Crime100 TAO RealTalk The Skin Deep (YouTube) SEUMLD Telecom Fraud Texts Emotional First Aid

Benchmarks: HumanSense

Metrics: accuracy average accuracy across tasks

Baselines: GPT-4o Qwen2.5-VL-7B Qwen2-VL-7B LLaVA-OneVision-7B LLaVA-NeXT-Video-7B VideoLLaMA3-7B InternVL3-8B Qwen2-Audio-7B Ola-7B IXC2.5-OmniLive-7B Qwen2.5-Omni-7B

Builds on: GRPO Qwen2.5-Omni HumanOmniV2

Closest papers by meaning

• HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenes (ICLR 2026 · similarity 0.90)
• LifeEval: A Multimodal Benchmark for Assistive AI in Egocentric Daily Life Tasks (CVPR 2026 · similarity 0.90)
• Video-Only ToM: Enhancing Theory of Mind in Multimodal Large Language Models (CVPR 2026 · similarity 0.89)
• InterMT: Multi-Turn Interleaved Preference Alignment with Human Feedback (NeurIPS 2025 · similarity 0.89)
• LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.88)
• Human-MME: A Holistic Evaluation Benchmark for Human-Centric Multimodal Large Language Models (ICLR 2026 · similarity 0.88)