Research map / Vision-language and multimodal / Multimodal LLMs

PGT: Procedurally Generated Tasks for improving visual grounding in MLLMs

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PGT, procedurally generated geometric-overlay QA tasks added to existing training images, which improve spatial, counting and 3D/depth understanding in MLLMs without degrading general perception.

Problem

MLLMs struggle with fine-grained spatial relational, quantitative and depth understanding, often relying on linguistic priors and dataset shortcuts rather than grounded visual geometry.

Approach

Overlay unambiguous geometric primitives (colored bounding boxes, semi-transparent circles, labeled points) on existing training images and procedurally generate QA pairs about them, so that the model must ground in visual geometry without language shortcuts; the overlay augments existing samples without increasing dataset size, and the skills transfer to real-world relational and 3D/depth benchma

Key results

• Instruction tuning Llama-3-8B with PGT raises What's Up from 77.8 to 97.4 (+19.6) and CV-Bench-2D from 58.5 to 71.8 (+13.3), with CV-Bench-3D up +9.7.
• Finetuning LLaVA-Next-7B with 5k PGT samples gives +8.3 on CV-Bench-2D, +6.2 on VSR and +14.9 on TallyQA.
• Finetuning Qwen2.5-VL-3B with PGT yields +7.8 on CV-3D-related depth gains, comparable to a +8.4 gain from a real-data Specialized Mix of the same size.

Surprising findings

• Skills learned from abstract 2D geometric overlays transfer to 3D/relative depth estimation on real images, with removing the distance task dropping 3D/Depth score from 70.9 to 61.7.
• Overlaying PGT on existing real images performs nearly as well as training on separate synthetic data while keeping the sample count unchanged.

Limitations

• Task suite is a proof-of-concept; authors acknowledge it is not exhaustive and extensions (3D primitives, temporal, multi-step) are left to appendix.
• Injecting more than two PGT tasks per image degrades 3D/depth performance and risks visual clutter.
• Gains on some general benchmarks are mixed (e.g., GQA and POPE slight declines for some backbones).

Experimental setup

Datasets: LLaVA-v1.5-Instruct Cambrian-7M TallyQA VSR Spatial-Ladder-26k

Benchmarks: What's Up CV-Bench-2D VSR CV-Bench-3D TallyQA MMVP RealWorldQA BLINK MMSTAR GQA POPE SEED

Metrics: accuracy

Baselines: LLaVA-Next-7B InternVL3-8B Qwen2.5-VL-3B Qwen2.5-VL-7B ViGoRL-3B Spatial-Ladder-3B ThinkLite-VL Specialized Mix

Builds on: LLaVA-v1.5 Prismatic training framework LoRA

Closest papers by meaning

• DiG: Differential Grounding for Enhancing Fine-Grained Perception in Multimodal Large Language Models (CVPR 2026 · similarity 0.91)
• GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation (CVPR 2026 · similarity 0.90)
• Perception Tokens Enhance Visual Reasoning in Multimodal Language Models (CVPR 2025 · similarity 0.90)
• Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.90)
• SIFThinker: Spatially-Aware Image Focus for Visual Reasoning (AAAI 2026 · similarity 0.90)
• Visual Jigsaw Post-Training Improves MLLMs (ICLR 2026 · similarity 0.90)