Research map / Vision-language and multimodal / Human and embodied modeling
Trimming the Long-Tail of Visual World Modeling Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces a benchmark testing whether image and video world models generalize physical interactions beyond common tool-task pairings, using Regular, Unconventional, and Impossible tool scenarios under predictive and descriptive generation.
Problem
Current world-model evaluations focus on common physical interactions, so it is unclear whether image and video generators internalize physical principles or rely on statistical co-occurrence in training data for rare or irregular tool-use interactions.
Approach
A three-mode scenario design (regular, attribute-compatible unconventional substitute, attribute-violating impossible tool) crossed with predictive vs descriptive prompting, scored with checklist rubrics by a VLM judge aligned with human annotation, to expose a long-tail performance gap.
Key results
- Across all evaluated image and video models, performance drops from Regular to Unconventional to Impossible scenarios; e.g., Nano-Banana-2 predictive Interaction Accuracy falls from 69% (regular) to 46% (unconventional) and 42% (impossible) in automatic scoring.
- Sora-2 is the best video model and Nano-Banana-2 the best image model under both automatic and human scoring.
- Video models score lower than image models on all metrics; e.g., Sora-2 predictive regular automatic IA 66% vs Nano-Banana-2 69%.
Surprising findings
- Descriptive prompts with explicitly specified outcomes often perform worse than predictive prompts, especially for video models, as models revert to familiar patterns.
- Models frequently ignore impossible-tool constraints and depict successful outcomes.
Limitations
- Benchmark is small: 80 tasks, 18 actions, with 1600 prompts total.
- Relies on a single VLM judge (gemini-2.5-pro) for automatic scoring, with human annotation by only nine annotators.
- Scope limited to tool-use interactions rather than multi-step embodied manipulation.
Experimental setup
Datasets: HICO-DET ConceptNet
Benchmarks: long-tail tool-use physical world modeling benchmark (introduced)
Metrics: Instruction Adherence Interaction Accuracy Physical Realism Perceptual Quality
Baselines: Z-Image Qwen-Image GPT-Image-1 Nano-Banana-2 HunyuanVideo-1.5 Wan-2.2 Sora-2 Veo-3.1
Builds on: HICO-DET ConceptNet VideoPhy MMGR
Closest papers by meaning
- WorldModelBench: Judging Video Generation Models As World Models (NeurIPS 2025 · similarity 0.90)
- Video World Models with Long-term Spatial Memory (NeurIPS 2025 · similarity 0.89)
- LongScape: Advancing Long-Horizon Embodied World Models with Context-Aware MoE (NeurIPS 2026 · similarity 0.88)
- World Models as Group Actions (NeurIPS 2026 · similarity 0.88)
- WorldReasonBench: Human-Aligned Stress Testing of Video Generators as Future World-State Predictors (NeurIPS 2026 · similarity 0.87)
- 4DWorldBench: A Comprehensive Evaluation Framework for 3D/4D World Generation Models (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.