Research map / Vision-language and multimodal / Multimodal LLMs
KiVA: Kid-inspired Visual Analogies for Testing Large Multimodal Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces KiVA, a developmentally inspired visual analogy benchmark of everyday object transformations, and compares LMMs (GPT-o1, GPT-4V, LLaVA-1.5, MANTIS) with human children and adults on verbal classification, specification, and visual extrapolation.
Problem
Existing visual analogy benchmarks use abstract shapes and grids with advanced skills, omitting basic real-world visual analogies that young children can solve, and they only score final answers rather than the reasoning steps.
Approach
A three-stage evaluation (verbal classification of what changed, verbal specification of how, and visual extrapolation to a new object) built on real-world 3D household objects with basic transformations (color, size, number, rotation, reflection), benchmarked against children aged 3-5 and adults.
Key results
- GPT-o1 identifies the transformation domain well but extrapolation declines sharply; it performs best on color and size and remains below child performance in reflection and number on KiVA.
- Children (ages 3-5, n=42) and adults show above-chance verbal classification, specification, and extrapolation, with adults near-perfect on KiVA.
- LLaVA-1.5 and MANTIS hallucinate a change in 100% of no-change trials during verbal classification and perform near chance on extrapolation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-o1 for visual extrapolation instead of other tested LMMs | GPT-o1, GPT-4V, LLaVA-1.5, and MANTIS · KiVA | visual extrapolation accuracy | Helped |
| Use LMMs instead of children for visual analogy tasks | GPT-o1, GPT-4V, LLaVA-1.5, and MANTIS · KiVA | visual extrapolation accuracy | Hurt |
| Use LMMs instead of adults for visual analogy tasks | GPT-o1, GPT-4V, LLaVA-1.5, and MANTIS · KiVA and KiVA-adults | visual extrapolation accuracy | Hurt |
| Reason about color and size transformations rather than number, rotation, and reflection | GPT-o1, GPT-4V, LLaVA-1.5, and MANTIS · KiVA | visual extrapolation accuracy | Helped |
| Add verbal classification and specification questions before visual extrapolation (ablation) | GPT-o1 · KiVA, number domain | visual extrapolation accuracy | No clear effect |
| Add verbal classification and specification questions before visual extrapolation (ablation) | GPT-o1 · KiVA, rotation domain | visual extrapolation accuracy | Hurt |
| Add verbal classification and specification questions before visual extrapolation (ablation) | GPT-o1 · KiVA, reflection domain | visual extrapolation accuracy | Helped |
| Present the given transformation and choices as four separate images instead of one combined image (ablation) | GPT-o1 · KiVA, color, size, and number domains | visual extrapolation accuracy | Helped |
| Present the given transformation and choices as four separate images instead of one combined image (ablation) | GPT-o1 · KiVA, rotation and reflection domains | visual extrapolation accuracy | No clear effect |
| Use text prompt engineering (reasoning through code, reflection, instructions, or in-context examples) (ablation) | Tested LMMs · KiVA | model performance | No clear effect |
| Alter image backgrounds or apply center crops as visual prompts (ablation) | LLaVA-1.5 · KiVA | model performance | No clear effect |
| Give the correct preceding verbal classification or specification answer (ablation) | Tested LMMs · KiVA, number, rotation, and reflection domains | subsequent task accuracy | No clear effect |
Surprising findings
- Prompt engineering (code reasoning, reflection, instruction, in-context learning) and visual prompting did not improve model performance.
- Providing correct verbal specification did not reliably improve visual extrapolation for models in weaker domains.
Limitations
- Open-source models were run on a single A6000 GPU for under 12 hours; closed-source GPT results depend on API access.
- Child sample is small (42 children on the main KiVA task), and only a random subset of 10 trials per child was used.
- Models were evaluated in multiple-choice format, which may not capture free-form reasoning.
Experimental setup
Datasets: 3D household object models toy object 3D models
Benchmarks: KiVA KiVA-adults
Metrics: accuracy response time choice inconsistency error score correlation
Baselines: GPT-4V LLaVA-1.5 MANTIS human children (ages 3-5) human adults
Builds on: Raven's Progressive Matrices ConceptARC MANTIS LLaVA-1.5
Closest papers by meaning
- ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generation (ICLR 2026 · similarity 0.89)
- BabyVision: Visual Reasoning Beyond Language (ICML 2026 · similarity 0.89)
- VOILA: Evaluation of MLLMs For Perceptual Understanding and Analogical Reasoning (ICLR 2025 · similarity 0.88)
- RBench-V: A Primary Assessment for Visual Reasoning Models with Multimodal Outputs (NeurIPS 2025 · similarity 0.88)
- When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thought (CVPR 2026 · similarity 0.88)
- MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.