Research map / Vision-language and multimodal / Multimodal LLMs

Human-MME: A Holistic Evaluation Benchmark for Human-Centric Multimodal Large Language Models

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Human-MME, a ~20K image-question benchmark across eight human-centric dimensions (face, body, HOI, multi-image, multi-person, intention, causal, emotion), built with an automated annotation pipeline plus expert review, and evaluates 17 MLLMs.

Problem

Existing MLLM benchmarks rarely evaluate fine-grained human-centric image understanding, lacking granular perception, grounding, multi-person/multi-image coverage, and high-level causal/intention/emotion reasoning with reliable annotations.

Approach

A progressive benchmark that moves from granular face/body/HOI perception to multi-person, multi-image, and causal/intention/emotion reasoning, with five question components (choice, short-answer, bounding box, ranking, judgment) and an automated annotation pipeline (YOLOv11, DWPose, Grounding DINO, FaceXFormer, Qwen2.5-VL-72B, Qwen3) verified by expert annotators.

Key results

• Gemini-2.5-Pro achieves the best closed-source average of 68.9 and GLM-4.5V the best open-source average of 76.0 across eight dimensions.
• Bounding-box performance is far lower than other components for most models, e.g., GLM-4.5V 66.3 IoU versus Gemini-2.5-Pro 23.5 and GPT-4o 11.5.
• Judgment tasks show high recall but low precision for most models, indicating hallucination tendency; Phi-4 has only 16.9% recall.

Surprising findings

• Intern-S1 and Gemini-2.5-Pro achieve strong high-level understanding scores despite weak fine-grained grounding.
• Adding a Judgment component to questions reduces performance for most models.

Limitations

• Automated annotation relies on Qwen2.5-VL-72B and Qwen3 outputs, creating potential same-model bias despite expert correction.
• Benchmark images are mostly from free media sites and HICO-DET, which may limit coverage of domains.
• Evaluation of Short-Answer uses a composite similarity score that may not fully capture answer correctness.

Experimental setup

Datasets: HICO-DET CelebA

Benchmarks: Human-MME

Metrics: accuracy IoU Kendall's Tau F1 composite short-answer score (BERT F1, embedding cosine simi precision recall

Baselines: GPT-4o Gemini-2.5-Pro Qwen2.5-VL-72B InternVL3-78B Intern-S1 GLM-4.5V

Builds on: HICO-DET CelebA DWPose Grounding DINO

Closest papers by meaning

• HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenes (ICLR 2026 · similarity 0.92)
• LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.91)
• MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models (ICLR 2025 · similarity 0.91)
• SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.91)
• L-Man: A Large Multi-modal Model Unifying Human-centric Tasks (AAAI 2025 · similarity 0.91)
• JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images (NeurIPS 2024 · similarity 0.91)