Research map / Vision-language and multimodal / Multimodal LLMs
Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Visual Program Distillation (VPD) samples multiple LLM-generated visual programs, keeps executable correct ones, converts their traces into chain-of-thought rationales, and distills them into PaLI-3 and PaLI-X so a VLM answers complex visual queries in one forward pass.
Problem
Visual programs built from LLM code generation and specialist vision tools solve compositional visual tasks but are error-prone, slow, and costly at inference, while standard visual instruction tuning lacks fine-grained compositional reasoning.
Approach
Use an LLM to sample k candidate ViperGPT-style programs, execute them with vision tools, filter for correct executions against labels, rewrite execution traces into natural-language CoTs, and fine-tune a VLM on both short answers and these rationales (step-by-step distillation).
Key results
- PaLI-X-VPD (55B) generalist achieves SOTA on all listed benchmarks, e.g. TallyQA complex 76.6 and MMBench 76.2 versus PaLI-X Instruct 75.4 and 75.0.
- Human raters find PaLI-X-VPD rationales factual 87.2% and consistent 97.8% of the time among samples with explanations, +14.6 and +10 points over PaLI-X Instruct.
- On Hateful Memes, PaLI-X-VPD specialist reaches 80.8% accuracy and 89.2 AUC-ROC, a new SOTA; the unsupervised variant with zero-shot CoTs also beats supervised VisualBERT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Distill verified visual programs as synthesized CoT instead of training without synthesized CoT (ablation) | PaLI-X (55B) · GQA | EM: +1.6 | Helped |
| Distill verified visual programs as synthesized CoT instead of training without synthesized CoT (ablation) | PaLI-X (55B) · TallyQA Complex | EM: +1.2 | Helped |
| Distill verified visual programs as synthesized CoT instead of training without synthesized CoT (ablation) | PaLI-X (55B) · MMBench | EM: +1.2 | Helped |
| Distill verified visual programs as synthesized CoT instead of training without synthesized CoT (ablation) | PaLI-X (55B) · POPE | EM | Hurt |
| Distill verified visual programs as synthesized CoT instead of training without synthesized CoT (ablation) | PaLI-X (55B) · VQA benchmarks | benchmark score | Mixed |
| Add CoT training to label-only fine-tuning (ablation) | PaLI-X-VPD on Hateful Memes · Hateful Memes seen test | Accuracy | Helped |
| Add CoT training to label-only fine-tuning (ablation) | PaLI-X-VPD on Hateful Memes · Hateful Memes seen test | AUC-ROC | Helped |
| Sample top-5 candidate programs instead of top-1 (ablation) | VPD program generation · GQA training data synthesis | program success rate: +45% | Helped |
| Sample top-5 candidate programs instead of top-1 (ablation) | VPD program generation · A-OKVQA training data synthesis | program success rate: +45% | Helped |
| Sample top-5 candidate programs instead of top-1 (ablation) | VPD program generation · OK-VQA training data synthesis | program success rate: +33% | Helped |
| Sample top-5 candidate programs instead of top-1 (ablation) | VPD program generation · TallyQA training data synthesis | program success rate: +10% | Helped |
| Use VPD instead of instruction tuning without synthesized CoT (ablation) | PaLI-X · Human evaluation | answer accuracy: +16.7% | Helped |
Surprising findings
- The distilled student (PaLI-X-VPD) is more accurate than the visual programs (teacher) it was distilled from on validation sets.
- Sampling 5 programs instead of 1 raises the success rate of finding a correct program by 10-45 points depending on dataset.
Limitations
- Programs from ViperGPT still fail on occluded objects or scenes with many objects.
- Unlabeled data cannot be filtered for program correctness; the unsupervised setting uses the top program without verification.
- The 3B UL2 language model in PaLI-3 seems too small to hold the knowledge needed for knowledge-based VQA.
Experimental setup
Datasets: VQAv2 OCR-VQA GQA OK-VQA A-OKVQA TallyQA Hateful Memes Visual Genome MMIT
Benchmarks: VQAv2 GQA OK-VQA A-OKVQA TallyQA TextVQA POPE MMBench Hateful Memes
Metrics: VQA score exact match accuracy AUC-ROC human correctness/factuality/consistency ratings
Baselines: PaLI-3 Instruct PaLI-X Instruct Flamingo InstructBLIP Shikra Qwen-VL LLaVA-1.5 VisualBERT Generated Programs
Builds on: ViperGPT Distilling step-by-step PaLI Program-of-Thoughts Self-Instruct
Closest papers by meaning
- Synthesizing Visual Concepts as Vision-Language Programs (CVPR 2026 · similarity 0.89)
- Long Grounded Thoughts: Synthesizing Visual Problems and Reasoning Chains at Scale (ICML 2026 · similarity 0.88)
- PRISM-Bench: A Benchmark of Puzzle-Based Visual Tasks with CoT Error Detection (NeurIPS 2026 · similarity 0.88)
- Don't Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs (CVPR 2026 · similarity 0.88)
- Thinking with Programming Vision: Towards a Unified View for Thinking with Images (CVPR 2026 · similarity 0.88)
- From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.