Research map / Vision-language and multimodal / Multimodal LLMs

Pointing at Parts: Training-Free Few-Shot Grounding in Multimodal LLMs

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes POP, a training-free plug-and-play method that fuses MLLM localization-head attention maps with bidirectional DINOv3 patch correspondences from few-shot support examples to enable part-level pointing.

Problem

Existing MLLMs point well at whole instances but struggle with part-level pointing, which is fine-grained, ambiguous, costly to annotate, and long-tailed.

Approach

Element-wise product of MLLM text-to-image localization-head attention maps with bidirectional (forward and backward) DINOv3 patch-similarity maps computed from a single support point per exemplar, requiring no training.

Key results

• Averaged over three datasets, POP improves Qwen2.5-VL-7B by 8.9 points in one-shot and 16.4 points in three-shot pointing accuracy.
• On non-pointing MLLMs, POP improves InternVL-3-8B and Kimi-VL-A3B by 25.3 and 30.9 points on average with one shot.
• On PACO with Molmo-7B-D, POP reaches 55.8 one-shot accuracy versus 51.2 zero-shot for the base model.

Surprising findings

• In-context learning with multiple support images can degrade pointing performance of MLLMs (e.g., Qwen2.5-VL-7B drops from 47.7 zero-shot to 14.1 one-shot on PACO).
• Chain-of-thought prompting can cause the model to point at the whole object instead of the part.

Limitations

• Performance on cluttered scenes with multiple objects (PACO) remains lower than for simpler datasets, and visual-only correspondences cannot disambiguate referring expressions.
• Relies on identifying localization heads per model, and the pipeline requires a labeled support point for each exemplar.

Experimental setup

Datasets: PACO-LVIS InstructPart PartImageNet++

Benchmarks: PACO-LVIS InstructPart PartImageNet++

Metrics: pointing accuracy (point inside ground-truth part mask)

Baselines: GPT-4.1 X-Decoder SEEM LISA VL-Part GF-SAM Matcher in-context learning chain-of-thought multi-turn

Builds on: Kang et al. localization heads Matcher F-LMM DINOv3

Closest papers by meaning

• Enhancing Part-Level Point Grounding for Any Open-Source MLLMs (CVPR 2026 · similarity 0.89)
• ClawMachine: Learning to Fetch Visual Tokens for Referential Comprehension (ICLR 2025 · similarity 0.88)
• Ferret: Refer and Ground Anything Anywhere at Any Granularity (ICLR 2024 · similarity 0.87)
• Coarse Correspondences Boost Spatial-Temporal Reasoning in Multimodal Language Model (CVPR 2025 · similarity 0.87)
• See&Trek: Training-Free Spatial Prompting for Multimodal Large Language Model (NeurIPS 2025 · similarity 0.87)
• Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception (CVPR 2024 · similarity 0.87)