Research map / Vision-language and multimodal / Multimodal LLMs

BOP-ASK: Object-Interaction Reasoning for Vision-Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces BOP-Ask, a ~33.8M-QA, ~150k-image dataset built from BOP 6D pose data for object-interaction reasoning (pose, grasps, trajectories, rearrangement, spatial and depth), plus BOP-Ask-core and BOP-Ask-lab benchmarks, and shows fine-tuned VLMs improve.

Problem

Vision-language models handle coarse spatial relations but lack fine-grained object-interaction reasoning needed for robotics: precise 3D localization, grasp affordances, collision-aware motion paths, and manipulation ordering in cluttered scenes.

Approach

Generate large-scale VQA annotations automatically from ground-truth 6D object poses in BOP datasets (world-frame reconstruction, RRT motion planning, M2T2 grasp prediction, and templated LLM-phrased questions), and fine-tune VLMs on them with pixel-level/3D outputs rather than multiple-choice questions.

Key results

• NVILA (15B) fine-tuned on BOP-Ask reaches 64.2% trajectory success rate and 57.7% rearrangement recall on BOP-Ask-core, versus 6.2% and 25.0% for base NVILA 15B.
• Qwen-VL 2.5 (3B) fine-tuned on BOP-Ask improves CV-Bench relation accuracy from 88.8 to 92.4 and lab spatial-depth accuracy from 74.4 to 85.8.
• On the real Franka robot, NVILA fine-tuned on BOP-Ask completed 10 of 15 pick-and-place tasks versus 0 for base NVILA.

Surprising findings

• Object rearrangement remains hard, with the best models reaching only about 60% accuracy.
• Base GPT-5 scored 0 success on trajectory prediction and produced invalid outputs for grasping.

Limitations

• Training data restricted to tabletop indoor scenes from BOP with about 104 objects.
• Trajectory and grasp prediction remain challenging, with low success rates on lab OOD data.
• Human baseline and test benchmarks are small (688 core VQAs, 240 lab VQAs).

Experimental setup

Datasets: HOPE HANDAL YCB-V LineMOD RoboSpatial-Home CV-Bench SpatialBench

Benchmarks: BOP-Ask-core BOP-Ask-lab RoboSpatial-Home CV-Bench SpatialBench

Metrics: 3D IoU success rate distance error normalized coordinate error recall accuracy

Baselines: GPT-5 Gemini Robotics-ER 1.5 Molmo Qwen-VL 2.5 NVILA RoboRefer Intern-VL RoboSpatial

Builds on: BOP M2T2 RRT NVILA Qwen-VL 2.5

Closest papers by meaning

• InternSpatial: A Comprehensive Dataset for Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.90)
• OBJECT-UNI: A Unified Model for Object-Centric Spatial Understanding and Controllable Generation (NeurIPS 2026 · similarity 0.90)
• SpinBench: Perspective and Rotation as a Lens on Spatial Reasoning in VLMs (ICLR 2026 · similarity 0.90)
• SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.89)
• SpatiaLab: Can Vision–Language Models Perform Spatial Reasoning in the Wild? (ICLR 2026 · similarity 0.89)
• Spatial Understanding from Videos: Structured Prompts Meet Simulation Data (NeurIPS 2025 · similarity 0.89)