Research map / 3D vision / 3D generation
The 3D-PC: a benchmark for visual perspective taking in humans and machines
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the 3D-PC benchmark, built from 3D Gaussian Splatting scenes rendered in Unity, to test depth order and visual perspective taking in humans and over 300 DNNs and VLMs.
Problem
Whether the emergent 3D analysis abilities of DNNs trained on static images suffice for visual perspective taking (VPT), a basic human social-cognitive ability, is unknown.
Approach
Generate counterbalanced depth-order and VPT labels on the same photorealistic 3D Gaussian Splatting scenes, and add a VPT-Strategy test with fixed camera and moving objects to probe whether models use line-of-sight or shortcut features.
Key results
- Humans were 86.82% accurate on VPT-basic, while the best linearly probed DNN (BEiT, ImageNet-21k) reached 53.82%; commercial VLMs were near chance (52%, 52%, 50%).
- DNN depth-order accuracy correlated with ImageNet accuracy (rho = 0.66, p < 0.001) after linear probing.
- After fine-tuning on VPT-basic, the best DNN (Swin Transformer) reached 66% on VPT-Strategy versus 87% for humans, and DNNs fell back toward chance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune DNNs on VPT-basic | TIMM DNNs · 3D-PC VPT-basic | accuracy | Helped |
| Fine-tune DNNs on VPT-basic | TIMM DNNs · 3D-PC VPT-Strategy | accuracy: 87% vs 66% | Hurt |
| Add chain-of-thought prompting | Vision-language models · 3D-PC VPT-basic | accuracy | No clear effect |
| Use linear probing or text-prompting instead of human judgment | DNNs and VLMs · 3D-PC depth order | accuracy | Mixed |
Surprising findings
- Nearly all DNNs matched or exceeded human accuracy on depth order yet were near chance on VPT-basic, the reverse of the human pattern.
- Fine-tuned DNNs reached near-human VPT-basic accuracy but failed to generalize to VPT-Strategy, suggesting shortcut strategies.
Limitations
- Could not isolate the monocular depth cues humans and DNNs use for depth order.
- VPT-basic is only the most basic form of VPT, far from general 3D reasoning.
- Small human sample sizes for VPT-Strategy (3 participants).
Experimental setup
Datasets: Common Objects in 3D (Co3D) ImageNet
Benchmarks: 3D-PC depth order 3D-PC VPT-basic 3D-PC VPT-Strategy
Metrics: accuracy Spearman correlation Cohen's kappa Dice similarity coefficient
Baselines: ChatGPT4 Gemini Claude 3 Stable Diffusion 2.0 Depth Anything Swin Transformer
Builds on: 3D Gaussian Splatting Common Objects in 3D (Co3D) Smooth gradients
Closest papers by meaning
- Evaluating Multiview Object Consistency in Humans and Image Models (NeurIPS 2024 · similarity 0.88)
- DepthCues: Evaluating Monocular Depth Perception in Large Vision Models (CVPR 2025 · similarity 0.88)
- VGGT: Visual Geometry Grounded Transformer (CVPR 2025 · similarity 0.88)
- ImageNet3D: Towards General-Purpose Object-Level 3D Understanding (NeurIPS 2024 · similarity 0.87)
- GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models (ICLR 2026 · similarity 0.87)
- Beyond Human Perception: Understanding Multi-Object World from Monocular View (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.