Research map / Vision-language and multimodal / Human and embodied modeling
Aligning MLLMs with the Latent Structure of Human Cognition via Behavior-Derived Semantic Dimensions
Ning E, Changde Du, Yizhuo Lu, Huiguang He
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes aligning MLLMs with the latent multidimensional structure underlying human similarity judgements by building on behavior-derived semantic dimensions.
Problem
Unclear whether MLLMs' internal representations and decision criteria align with the latent structure of human cognition, particularly the multidimensional mental representations underlying similarity judgements.
Approach
A framework that aligns MLLM representations with behavior-derived semantic dimensions recovered from human similarity judgements, building on a previous (unspecified) line of work on such dimensions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune on dimension-grounded rationales (CogAlign) instead of CoT prompting | Qwen3-VL-4B · O1O human consistency | accuracy (%): 53.37 -> 56.81 | Improved |
| CogAlign versus Direct prompt baseline | Qwen3-VL-30B-A3B · O1O human consistency | accuracy (%): 37.09 -> 56.21 | Improved |
| fine-tune with CogAlign versus CoT baseline as model size grows | Qwen3-VL (4B, 8B, 30B-A3B) · O1O human consistency | accuracy (%): 53.37 -> 56.81; 52.28 -> 56.99; 55.38 -> 56.21 | Improved |
| add CogAlign fine-tuning | Qwen3-VL · MMMU val | accuracy (%): 53.22 -> 57.56 (30B-A3B) | Improved |
| add CogAlign fine-tuning | Qwen3-VL · MMBench en test | accuracy (%): 87.44 -> 83.86 (30B-A3B); 82.96 -> 81.39 (4B) | Hurt |
| constrain inference-time CoT to the top-1 salient dimension d* (ablation) | Qwen3-VL-4B · O1O human consistency | accuracy (%): 53.37 -> 49.01 | Hurt |
| prompt with full 66-dimension space at inference (FS-CoT) (ablation) | Qwen3-VL-4B · O1O human consistency | accuracy (%): 53.37 -> 52.70 | No clear effect |
| apply CogAlign to other model families (ablation) | Qwen2.5-VL-7B and Gemma3-12B · O1O validation subset | accuracy (%): 56.63 -> 59.29 (Qwen2.5-VL); 55.77 -> 57.28 (Gemma3) | Improved |
| use rationales generated by DeepSeek-V4-Flash instead of Qwen2.5-72B (ablation) | CogAlign Qwen3-VL-4B · O1O human consistency | accuracy (%): 56.81 -> 56.52 | No clear effect |
| align Qwen2.5-VL-7B with CogAlign for behavioral geometry | Qwen2.5-VL-7B · 48-object choice-based RSM correlation | correlation (%): 73.30 -> 80.93 | Improved |
Closest papers by meaning
- From Black Boxes to Transparent Minds: Evaluating and Enhancing the Theory of Mind in Multimodal Large Language Models (ICML 2025 · similarity 0.89)
- Human-MME: A Holistic Evaluation Benchmark for Human-Centric Multimodal Large Language Models (ICLR 2026 · similarity 0.88)
- The Alignment Illusion in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.88)
- LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.88)
- Turning large language models into cognitive models (ICLR 2024 · similarity 0.88)
- The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
