Research map / Reinforcement learning / Imitation learning

SyncWorld: Visual Calibration Enables World Models as Zero-Shot Simulators

NeurIPS 2026 · Reinforcement learning › Imitation learning · approach: Imitation combined with RL · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SyncWorld, an action-conditioned video world model (fine-tuned Wan2.2-TI2V-5B) that conditions on a short visual calibration episode and distills calibration into history-only inference, enabling zero-shot simulation across unseen cameras and embodiments and test-time policy ranking.

Problem

Action-conditioned world models fail to generalize when camera view, robot placement, or embodiment changes, because the same numerical action produces different pixel motion (setup-dependent Action-Visual Mapping), causing conflicting supervision and brittle deployment.

Approach

Provide an in-context visual calibration episode (paired frames and actions covering all six motion DoFs) as a prefix condition so the world model infers the setup-specific action-to-visual mapping; train with action-coordinate augmentation and calibration-to-history distillation so the model also works without explicit calibration.

Key results

• SyncWorld w/ Calib reaches PSNR 28.3 / LPIPS 0.035 / FID 7.0 on LIBERO, versus best baseline Ctrl-World at 24.8 / 0.137 / 16.5.
• Cross-view Met3R average 0.538 with calibration vs 0.565 for Ctrl-World and 0.560 average-ground-truth oracle gap of 0.523.
• On three oracle-headroom LIBERO tasks, SyncWorld w/ Calib improves success over the pi0 baseline (e.g., BBQ Sauce 0.58 vs 0.52, Orange Juice 0.72 vs 0.56), approaching oracle (0.60, 0.80).

Surprising findings

• Without test-time calibration, history-only SyncWorld performs close to the calibrated variant on most metrics.
• On some LIBERO tasks (Ketchup, Put Cream Cheese), oracle ground-truth rollouts with GPC-Rank perform no better or worse than the direct policy, indicating the VLM ranking is the bottleneck.

Limitations

• Failures under extreme camera views with gripper dominating the image, yielding ambiguous motion predictions.
• Failures with unseen object interaction; only single-arm embodiments tested.
• Policy improvement evaluated only on oracle-headroom LIBERO subset; gains depend on VLM ranking quality.

Experimental setup

Datasets: RLBench RoboCasa RoboMimic DROID LIBERO ManiSkill

Benchmarks: LIBERO ManiSkill Real (xArm) self-collected

Metrics: PSNR SSIM LPIPS FID Met3R success rate

Baselines: IRASim WorldGym Ctrl-World Oracle simulator (LIBERO) Direct policy (pi0)

Builds on: Wan2.2 GPC-Rank Ctrl-World IRASim WorldGym

Closest papers by meaning

• Beyond World-Frame Action Heads: Motion-Centric Action Frames for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.89)
• EA-WM: Event-Aware Generative World Model with Structured Kinematic-to-Visual Action Fields (NeurIPS 2026 · similarity 0.89)
• Action Images: End-to-End Policy Learning via Multiview Video Generation (NeurIPS 2026 · similarity 0.89)
• DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning (ICML 2025 · similarity 0.88)
• OSVI-WM: One-Shot Visual Imitation for Unseen Tasks using World-Model-Guided Trajectory Generation (NeurIPS 2025 · similarity 0.88)
• Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation (ICLR 2025 · similarity 0.88)