Research map / Vision-language and multimodal / Multimodal LLMs

Thinking with Camera: A Unified Multimodal Model for Camera-Centric Understanding and Generation

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Puffin is a unified camera-centric multimodal model that performs camera calibration (understanding) and camera-controllable text-to-image generation via a "thinking with camera" chain-of-thought, trained on the 4M-triplet Puffin-4M dataset.

Problem

Camera-centric understanding (estimating roll, pitch, FoV) and camera-controllable generation are typically studied in isolation, and existing LMMs ignore or misinterpret numerical camera parameters, producing spatially inconsistent results.

Approach

Treat the camera as language: map numerical roll/pitch/FoV to photographic terms and generate spatial-reasoning captions, so an LMM reasons over spatially grounded cues before predicting or rendering camera geometry; pair this with pixel-wise perspective-field camera maps as continuous latent conditions for a diffusion model in a shared LLM+diffusion framework.

Key results

• On MegaDepth, Puffin achieves roll error 0.32° and pitch error 1.08° versus GeoCalib's 0.36° and 1.94°, and best AUC on roll and pitch.
• On Puffin-Gen, Puffin reduces median up-vector error to 10.12° versus 17.47° for PreciseCam and 22.10° for GPT-4o, and reaches FID 69.46 versus 83.31 for Qwen-Image.
• Puffin-Und results show roll error 0.41° and pitch error 0.74°, outperforming GeoCalib (0.92°, 2.18°); Puffin slightly underperforms GeoCalib on LaMAR FoV (3.62° vs 3.03°) and TartanAir FoV.

Surprising findings

• Re-training GeoCalib on the 4M Puffin-4M dataset slightly underperforms its original 40K-trained version, which the authors attribute to limited model capacity.
• Existing image generators are worse at simulating roll than pitch, the reverse of the trend seen in camera understanding, attributed to dataset bias toward near-level shots.

Limitations

• Fixed 512x512 training resolution limits generation to a single scale, and central cropping for non-square understanding inputs can discard content and hurt performance.
• Generation evaluation relies on an offline vision-based calibration method whose errors may be ambiguous for subtle differences.
• Only ideal pinhole camera model is used; radial distortion is left as future work.

Experimental setup

Datasets: Puffin-4M Stanford2D3D MegaDepth TartanAir LaMAR

Benchmarks: MegaDepth TartanAir LaMAR Puffin-Und Puffin-Gen Stanford2D3D

Metrics: median error AUC at 1/5/10 degrees up vector error latitude error gravity error FID

Baselines: DeepCalib Perceptual CTRL-C MSCC ParamNet SVA UVP GeoCalib PreciseCam GPT-4o Qwen-Image Nano Banana

Builds on: GeoCalib PreciseCam SD3 Perspective Fields C-RADIO

Closest papers by meaning

• SpaceMind: Camera-Guided Modality Fusion for Spatial Reasoning in Vision-Language Models (CVPR 2026 · similarity 0.90)
• On the Generalization Capacities of MLLMs for Spatial Intelligence (ICLR 2026 · similarity 0.89)
• Exploring Spatial Intelligence from a Generative Perspective (CVPR 2026 · similarity 0.89)
• From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.89)
• From Correspondence to Actions: Human-Like Multi-Image Spatial Reasoning in Multi-modal Large Language Models (ICML 2026 · similarity 0.89)
• Thinking with Imagination: Agentic Visual Spatial Reasoning with World Simulators (NeurIPS 2026 · similarity 0.89)