Research map / Vision-language and multimodal / Multimodal LLMs

Spot The Ball: A Benchmark for Visual Social Inference

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Spot the Ball, a benchmark where VLMs must localize a removed ball in soccer, basketball, and volleyball images using player gaze and pose cues, finding humans 2-3x more accurate than four VLMs.

Problem

Whether vision-language models can infer hidden objects in a scene from visual social cues such as player gaze, pose, and orientation, as humans do.

Approach

Uses a classic newspaper puzzle format on sports images with the ball inpainted out, so localization must rely on social cues from players; compares VLM predictions against human participants under three prompting strategies and analyzes spatial heuristics (center bias, player proximity) and textual reasoning about gaze vs pose.

Key results

• Human accuracy ranges 19-34% across sports while all four VLMs remain at or below 17%.
• In volleyball, humans have mean Euclidean error 72.0 px versus about 150-273 px for models.
• Models predict near a player about 90% of the time versus 65-75% for humans; humans show higher normalized entropy (0.855) than models (0.698-0.808).

Surprising findings

• Chain-of-thought prompting degraded accuracy for some models (e.g., LLaMA and Qwen in volleyball, error 272.6 px), and gains from cue-directed prompting were inconsistent.
• Models mention pose cues far more than gaze in their reasoning, even though gaze is the cue humans use relatively evenly.

Limitations

• Evaluation set is small (150 images) and human baselines are from a single pre-registered crowd sample per sport.
• Proprietary model internals undisclosed, limiting architectural attribution.
• Static images only; the paper does not test temporal or motion cues.

Experimental setup

Datasets: YouTube soccer, basketball, volleyball broadcast footage

Benchmarks: Spot the Ball

Metrics: accuracy mean Euclidean error Wasserstein distance near player rate overlap rate center ratio normalized entropy

Baselines: Human participants (Prolific) Base prompt Cue-directed prompt Chain-of-thought prompt Uniform distribution baseline

Builds on: chain-of-thought prompting YOLOv8 CLIP Stable Diffusion inpainting

Closest papers by meaning

• Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models (NeurIPS 2026 · similarity 0.88)
• SportR: A Benchmark for Multimodal Large Language Model Reasoning in Sports (ICLR 2026 · similarity 0.87)
• Behavioral Geometric Supervision Aligns Video Foundation Models with Human Social Perception (NeurIPS 2026 · similarity 0.87)
• HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenes (ICLR 2026 · similarity 0.86)
• SoMi-ToM: Evaluating Multi-Perspective Theory of Mind in Embodied Social Interactions (NeurIPS 2025 · similarity 0.85)
• OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following (NeurIPS 2026 · similarity 0.85)