Research map / Vision-language and multimodal / Human and embodied modeling
DevBench: A multimodal developmental benchmark for language learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DevBench, a seven-task multimodal language benchmark with child and adult response data, comparing vision-language models to humans via a softmax-optimised KL divergence and RSA, and tracks OpenCLIP-H checkpoints over training.
Problem
Existing multimodal and developmental benchmarks evaluate vision-language models against adult-level accuracy without direct comparison to child behavioral data, limiting insight into whether models learn language like children do.
Approach
Evaluate models on lexical, syntactic, and semantic tasks with item-level human response distributions from children and adults, and score models by similarity of their response patterns to humans (softmax-optimised KL divergence for choice tasks, RSA for representational tasks) rather than by accuracy.
Key results
- Across tasks, CLIP-large and OpenCLIP-H performed relatively well on lexicon and syntax tasks; CVCL was mostly dissimilar to humans on lexicon and syntax tasks (e.g., VV KL 0.740 vs human 0.091).
- Model-human similarity correlated most strongly with task accuracy across tasks (e.g., r = 0.998 for adult VV), with size next and training examples weakest.
- OpenCLIP-H became more human-like over training for VV, TROG, and WG, while similarity to THINGS decreased with training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase OpenCLIP-H training | OpenCLIP-H · VV, TROG, and WG | model-human similarity | Helped |
| Increase OpenCLIP-H training | OpenCLIP-H · LWL | model-human similarity | Helped |
| Increase OpenCLIP-H training | OpenCLIP-H · WAT | model-human similarity | No clear effect |
| Increase OpenCLIP-H training | OpenCLIP-H · VOC | model-human similarity | Mixed |
| Increase OpenCLIP-H training | OpenCLIP-H · THINGS | model-human similarity | Hurt |
| Higher task accuracy | Vision-language models · Visual Vocabulary (VV) | model-human similarity: b~=-0.057, SE = 0.003, p < .001 | Helped |
| Train CVCL on child head-mounted-camera data | CVCL · VOC | RSA similarity: 0.138 | Helped |
| Use next-token prediction rather than log-likelihood measurement | TinyLLaVA · VV accuracy | accuracy: 0.462 -> 0.958 | Helped |
| Use next-token prediction rather than log-likelihood measurement | TinyLLaVA · TROG accuracy | accuracy: 0.474 -> 0.782 | Helped |
| Use next-token prediction rather than log-likelihood measurement | TinyLLaVA · LWL accuracy | accuracy: 0.855 -> 1.000 | Helped |
| Use next-token prediction rather than log-likelihood measurement | TinyLLaVA · WG accuracy | accuracy: 0.614 -> 0.728 | Helped |
Surprising findings
- Models' human-likeness on THINGS decreased over training, suggesting multimodal contrastive learning may yield less human-like visual representations.
- SigLIP was unusually poorly aligned with human responses despite high accuracy and was excluded as an outlier.
Limitations
- Human datasets are small for some tasks (few items and participants), leading to uncertain reliability.
- Tasks are English-only and mainly from English-speaking children and adults.
- Model-feature correlations cannot disentangle architecture from size or training data because architectures could not be held constant.
Experimental setup
Datasets: Looking-while-listening datasets (Adams, Frank, Donnelly & Kidd) Visual vocabulary task data (Long et al.) TROG stimuli and data (Silverman & Yeatman) Winoground NoTag Entwisle word association Nelson et al. word association norms Spriet et al. infant VOC data THINGS odd-one-out data
Benchmarks: DevBench LWL DevBench VV DevBench TROG DevBench WG DevBench WAT DevBench VOC DevBench THINGS
Metrics: softmax-optimised KL divergence RSA Spearman correlation accuracy Pearson correlation with features
Baselines: CLIP-base CLIP-large ViLT FLAVA BLIP BridgeTower OpenCLIP-H SigLIP CVCL LLaVA-NeXT TinyLLaVA Kosmos-2
Builds on: CLIP Looking-while-listening Winoground THINGS RSA
Closest papers by meaning
- BabyVLM-V2: Toward Developmentally Grounded Pretraining and Benchmarking of Vision Foundation Models (CVPR 2026 · similarity 0.88)
- MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks (ICLR 2026 · similarity 0.87)
- CogBench: a large language model walks into a psychology lab (ICML 2024 · similarity 0.87)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.87)
- OmniBench: Towards The Future of Universal Omni-Language Models (NeurIPS 2025 · similarity 0.87)
- BEAR: Dissecting Embodied Abilities in Multimodal Language Models through Skill-level Evaluation and Diagnosis (ICML 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.