Research map / Vision-language and multimodal / Multimodal LLMs

HSSBench: Benchmarking Humanities and Social Sciences Ability for Multimodal Large Language Models

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces HSSBench, a 13,152-item multilingual multiple-choice and open-ended VQA benchmark for humanities and social sciences, built with an expert plus multi-agent generation pipeline, and evaluates over 20 MLLMs.

Problem

Existing MLLM benchmarks emphasize general knowledge and STEM vertical reasoning, leaving humanities and social sciences (HSS) tasks that need horizontal, interdisciplinary reasoning and linking abstract concepts to images underexplored.

Approach

A VQA generation pipeline in which domain experts and GPT-4o/GPT-4.1 agents (summarizer, extractor, question generator, image matcher, validator) collaboratively create HSS questions, with validation requiring that neither text alone nor image alone suffices, followed by translation into six UN languages.

Key results

• GPT-4.1 reaches 42.66% on multiple-choice and 39.97% on open-ended questions in the English setting, while the best open-source model Qwen2.5-VL-72B reaches 51.87% on multiple-choice but only 19.73% open-ended.
• Human experts average 93.83% overall accuracy, far above the best models.
• Most models score below 60% on HSSBench, with economics the hardest category and geography the easiest.

Surprising findings

• Chain-of-thought prompting does not consistently help and sometimes lowers accuracy for several models.
• RAG with Wikipedia and HSS documents did not consistently improve accuracy over direct prompting.

Limitations

• Multiple-choice format dominates; only a subset of questions converted to open-ended.
• Expert pool is mostly Chinese-speaking, which may introduce cultural bias that the authors suggest may explain Qwen's strong showing on Culture and Social Sciences.
• Data generation relies heavily on GPT-4o/GPT-4.1 agents, and validation is partly agent-driven.

Experimental setup

Datasets: HSSBench

Benchmarks: HSSBench

Metrics: accuracy

Baselines: GPT-4o GPT-4.1 GPT-4.1-mini GPT-4.1-nano Qwen2.5-VL-72B-Instruct Qwen2-VL-72B-Instruct QVQ-72B-Preview InternVL3-8B InternVL2.5-8B-MPO Llava-onevision-7b

Builds on: chain-of-thought prompting retrieval-augmented generation

Closest papers by meaning

• MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models (AAAI 2026 · similarity 0.91)
• SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.91)
• SO-Bench: A Structural Output Evaluation of Multimodal LLM (CVPR 2026 · similarity 0.90)
• MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models (AAAI 2026 · similarity 0.90)
• LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.89)
• RBench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation (ICML 2025 · similarity 0.89)