Research map / Vision-language and multimodal / Human and embodied modeling
Testing the Limits of Fine-Tuning for Improving Visual Cognition in Vision Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes Qwen2-VL-7B and Llama 3.2 11B/90B Vision with QLoRA on synthetic Cubeworld block-tower stability and pyramid counterfactual data, finding in-domain and size-shift gains but failure to transfer across visual style or cognitive domain.
Problem
Whether task-specific fine-tuning of vision language models can give robust, human-like intuitive physics and causal reasoning that generalizes to new tower sizes, visual styles, and cognitive domains.
Approach
Builds a controlled synthetic dataset (Cubeworld) of block towers and pyramids paired with ground truth and human judgments, then systematically tests generalization of QLoRA-fine-tuned VLMs across four conditions: same-distribution, taller towers, real images, and the other cognitive domain.
Key results
- Fine-tuned 11B models reach accuracies of 0.6 to 0.92 on single block sizes from their fine-tuning split, versus near-chance zero-shot base models.
- Models fine-tuned on 2-4 block towers still beat the base model on taller towers, though accuracy drops as block count grows.
- Models fine-tuned on one cognitive domain do not transfer to the other, while joint models perform well on both; fine-tuning on human judgments raised mean kappa with humans on the same task to 0.37.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune on ground-truth intuitive physics data | Qwen2-VL-7B, Llama 3.2-11B, and Llama 3.2-90B · Cubeworld intuitive physics, training-distribution towers | accuracy: between $0.78$ and $0.83$ | Helped |
| Fine-tune on ground-truth causal reasoning data | Vision-language models · Cubeworld causal reasoning pyramids | accuracy | Helped |
| Fine-tune on 2–4-block intuitive physics towers | Vision-language models · Lerer naturalistic block towers | accuracy | Mixed |
| Fine-tune on one cognitive domain rather than both | Vision-language models · The other Cubeworld cognitive domain | accuracy | Hurt |
| Fine-tune on data from both intuitive physics and causal reasoning | Vision-language models · Cubeworld intuitive physics and causal reasoning | accuracy | Helped |
| Fine-tune on ground-truth 2–4-block intuitive physics data | Llama 3.2-11B · Cubeworld intuitive physics, same task | Cohen's κ with humans: $0.23$ | Helped |
| Fine-tune on ground-truth 3- and 6-block causal reasoning data | Llama 3.2-11B · Cubeworld causal reasoning, same task | Cohen's κ with humans: $0.4$ | Helped |
| Fine-tune on human responses instead of ground-truth labels | Llama 3.2-11B · Cubeworld intuitive physics, same task | Cohen's κ with humans: $0.23$ -> $0.37$ | Helped |
| Fine-tune on human responses instead of ground-truth labels | Llama 3.2-11B · Lerer naturalistic block towers | Cohen's κ with humans: $0.15$ -> $0.37$ | Helped |
| Fine-tune on human responses for intuitive physics | Llama 3.2-11B · Cubeworld causal reasoning pyramids | Cohen's κ with humans: $-0.02$ | No clear effect |
| Train on 2–4-block towers rather than larger towers | Vision-language models fine-tuned for intuitive physics · Cubeworld intuitive physics, increasing tower size | accuracy | Mixed |
| Fine-tune on human intuitive-physics judgments rather than ground truth | Vision-language models · Cubeworld intuitive physics, ground-truth stability prediction | accuracy | Helped |
Surprising findings
- 11B model fine-tuned on 2-4 block Cubeworld towers performs only above chance on 4-block Lerer towers, with some sizes falling below chance.
- Causal-reasoning fine-tuning does not transfer to intuitive physics even on paired images where the counterfactual task requires the stability judgment as a prerequisite.
Limitations
- Only models up to 90B parameters were tested; closed-source larger models were not fine-tuned.
- Visual and cognitive demands are entangled, so it is unclear which factor drives failures.
- Fine-tuning used a single PEFT configuration (r=16, alpha=16) and only supervised learning; RL-based post-training was not tested.
Experimental setup
Datasets: Cubeworld intuitive physics towers Cubeworld causal reasoning pyramids Lerer et al. (2016) naturalistic block towers human judgments collected on Prolific
Benchmarks: Cubeworld intuitive physics evaluation set Cubeworld causal reasoning evaluation set Lerer et al. (2016) naturalistic block towers
Metrics: accuracy Cohen's kappa (bootstrapped) error consistency
Baselines: zero-shot base models
Builds on: QLoRA PaliGemma-3B unsloth
Closest papers by meaning
- VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models (CVPR 2026 · similarity 0.90)
- From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models (ICML 2026 · similarity 0.90)
- Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.90)
- On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMs (ICML 2026 · similarity 0.89)
- Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Models (NeurIPS 2025 · similarity 0.89)
- Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.