Research map / Vision-language and multimodal / Multimodal LLMs

Enhancing Part-Level Point Grounding for Any Open-Source MLLMs

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Equips frozen open-source MLLMs with accurate 2D part-level point grounding by learning to synthesize grounding-aware queries from attention heads (Q-Synth) and decoding a high-resolution point heatmap (A2P) trained with BCE and an SDF-based penalty field.

Problem

MLLMs excel at object-level point grounding but struggle with part-level point grounding needed for fine-grained tasks like robotic manipulation; native text-to-image attention is imprecise and fine-tuning for grounding risks degrading the model's general abilities.

Approach

Keep all MLLM parameters frozen; learn a Q-Synth module that aggregates all text-query features via latent cross-attention into one grounding-aware query per localization head, producing refined attention maps, and an A2P decoder that fuses these with image value features and upsamples to a point heatmap supervised by an asymmetric SDF-based penalty field.

Key results

• On PACO, Molmo-7B with the method reaches 0.510 accuracy vs 0.487 for its text pointing and 0.428 for its attention pointing.
• On InstructPart, the method lifts LLaVA-1.5-7B from 0.194 (attention pointing) to 0.783 accuracy.
• On PointArena Affordance, Qwen2.5-VL-7B with the method reaches 0.828 accuracy vs 0.303 for its attention pointing when trained only on PACO.

Surprising findings

• The method, with frozen weights, gives a non-pointing model (LLaVA-1.5-7B) accuracy comparable to native pointing models on PACO and InstructPart.
• Latency is lower than autoregressive text pointing (118.59 ms vs 588.22 ms for LLaVA-1.5-7B) because only one forward pass is needed.

Limitations

• Outputs only a single point per query; multi-target scenes are not supported.
• Does not handle images where no target is present.
• Evaluated mainly on PACO and InstructPart; generalization limited to the PointArena tasks reported.

Experimental setup

Datasets: PACO InstructPart

Benchmarks: PACO InstructPart PointArena Point-Bench

Metrics: accuracy (hit-or-miss) patch accuracy

Baselines: Molmo-7B text pointing Molmo-7B attention pointing Qwen2.5-VL-7B text pointing Qwen2.5-VL-7B attention pointing LLaVA-1.5-7B attention pointing VLPart X-Decoder LISA-7B GLaMM-7B

Builds on: localization heads (zero-shot attention grounding) Molmo LLaVA-1.5

Closest papers by meaning

• GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation (CVPR 2026 · similarity 0.90)
• GROUNDHOG: Grounding Large Language Models to Holistic Segmentation (CVPR 2024 · similarity 0.90)
• Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding (CVPR 2025 · similarity 0.90)
• Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning (AAAI 2026 · similarity 0.90)
• Referencing Where to Focus: Improving Visual Grounding with Referential Query (NeurIPS 2024 · similarity 0.89)
• Pointing at Parts: Training-Free Few-Shot Grounding in Multimodal LLMs (CVPR 2026 · similarity 0.89)