Research map / Vision-language and multimodal / Multimodal LLMs

Synthesizing Visual Concepts as Vision-Language Programs

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Vision-Language Programs (VLP), which use VLMs to ground symbols and extract structured visual descriptions that are compiled into a PCFG-based DSL and searched for programs that induce visual concepts from few labeled images.

Problem

VLMs fail at systematic inductive visual reasoning (inducing rules that separate positive from negative image sets), while existing neuro-symbolic methods rely on rigid domain-specific perception modules or explicit queries.

Approach

Decouple perception from reasoning: the VLM is used only as a set of callable perception functions (e.g., get_objects, get_actions) inside a typed DSL; a PCFG prior with occurrence-based symbol weighting guides program search over labeled support images, yielding training-free, executable, interpretable rules.

Key results

• VLP raises InternVL3-8B average balanced accuracy from 57.4% to 70.9% across five datasets, and Qwen2.5-VL-7B on CLEVR-Hans3 from 46.1% to 86.1%.
• With VLP, Qwen3-VL-30B-A3B reaches 66.1% on CLEVR-Hans3 (+16.1) and GPT-5 with low-effort VLP uses far fewer tokens than GPT-5 thinking (e.g., 8,058 vs 98,046 on CLEVR-Hans3).
• Removing shortcut colors from the DSL improves Qwen3-VL VLP performance by 13.3% on CLEVR-Hans3.

Surprising findings

• VLP underperforms direct prompting on Bongard-OpenWorld, attributed to annotation errors in the dataset.
• Feeding the structured VLM-function outputs directly to the VLM does not reliably help, indicating gains come mainly from symbolic search rather than the representation format.

Limitations

• Performance limited by VLM perception errors such as malformed outputs (e.g., ~13% of Kimi COCOLogic outputs) and omitted task-relevant properties.
• Lacks explicit object representations, limiting object-centric concepts such as spatial relations.
• Annotation errors in Bongard-OpenWorld cause VLP to fail on query samples.

Experimental setup

Datasets: Bongard-HOI Bongard-OpenWorld Bongard-RWR COCOLogic CLEVR-Hans3

Benchmarks: Bongard-HOI Bongard-OpenWorld Bongard-RWR COCOLogic CLEVR-Hans3

Metrics: balanced accuracy tokens

Baselines: InternVL3-8B InternVL3-14B Kimi-VL-A3B-Instruct Qwen2.5-VL-7B-Instruct Qwen3-VL-30B-A3B-Instruct Kimi-VL-A3B-Thinking Qwen3-VL-30B-A3B-Thinking gpt-5

Builds on: VisProg ViperGPT NePTune Heap Search Algorithm

Closest papers by meaning

• From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models (ICML 2026 · similarity 0.92)
• VLMs have Tunnel Vision: Evaluating Nonlocal Visual Reasoning in Leading VLMs (NeurIPS 2025 · similarity 0.91)
• NePTune: A Neuro-Pythonic Framework for Tunable Compositional Reasoning on Vision-Language (ICLR 2026 · similarity 0.91)
• Don't Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs (CVPR 2026 · similarity 0.91)
• Perception Tokens Enhance Visual Reasoning in Multimodal Language Models (CVPR 2025 · similarity 0.90)
• Concept-RuleNet: Grounded Multi-Agent Neurosymbolic Reasoning in Vision Language Models (AAAI 2026 · similarity 0.90)