Research map / Video / Video-language understanding
Apple-$\pi$: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Apple-π, a law-grounded benchmark (400-video Orchard mechanics dataset, three-stage Perception/Formulation/Deduction chain-of-frames protocol, hybrid MLLM and physics-grounded metrics) for testing whether video and unified models reason physically, evaluating 11 models.
Problem
Existing video-generation physics benchmarks evaluate only output plausibility, not whether models reach outputs through law-grounded reasoning, so it is unclear whether video models internalize physical laws.
Approach
Anchors video-model evaluation in classical-mechanics laws using infographic-annotated first frames and chain-of-frames prompts, splitting evaluation into Perception, Formulation, and Deduction stages with stage-resolved hybrid metrics (MLLM judge plus physics-grounded IoU and trajectory error).
Key results
- Best video model Seedance 2.0 scores 0.473 average, while GPT Image 2 and Nano Banana 2 reach 0.704 and 0.699.
- Deduction is the hardest stage; even the strongest unified models score only around 0.40 on Deduction.
- Video and unified models both score lower on real-world than simulated cases, indicating a Sim-to-Real gap.
Surprising findings
- Unified image-generation models (GPT Image 2, Nano Banana 2) substantially outperform video models on the benchmark average.
- Text-parameter prompting of physical quantities is nearly neutral versus infographic overlays.
Limitations
- Covers only ten classical-mechanics task families with four primitive object shapes.
- Deduction and Formulation scores depend on an MLLM judge (Gemini 3 Flash) for subjective criteria.
Experimental setup
Datasets: Orchard
Benchmarks: Apple-π
Metrics: MLLM subjective score segmentation IoU pixel-level fidelity spatiotemporal mask overlap 3D velocity error
Baselines: Wan2.2 HunyuanVideo-1.5 VBVR-Wan2.2 Veo 3.1 BAGEL OmniGen2 SenseNova-U1-8B-MoT
Builds on: chain-of-frames VideoPhy Physics-IQ
Closest papers by meaning
- SciVideoBench: Benchmarking Scientific Video Reasoning in Large Multimodal Models (ICML 2026 · similarity 0.87)
- Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation (ICML 2025 · similarity 0.87)
- How Far Is Video Generation from World Model: A Physical Law Perspective (ICML 2025 · similarity 0.87)
- QUANTIPHY: A Quantitative Benchmark Evaluating Physical Reasoning Abilities of Vision-Language Models (CVPR 2026 · similarity 0.87)
- VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understanding (NeurIPS 2025 · similarity 0.86)
- Principia: Relational Physics Tests for Video Models (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.