Research map / Image generation / Editing and style transfer
PhysicsGen: Can Generative Models Learn from Images to Predict Complex Physical Relations?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PhysicsGen, a 300k image-pair benchmark testing whether generative image-to-image models (GANs, VAEs, U-Nets, diffusion) can learn three physical simulations (sound propagation, lens distortion, bouncing ball dynamics), and measures speedups and physical accuracy.
Problem
Whether generative image-to-image models can learn complex physical relations from input-output image pairs with physical correctness, and what speedups they offer over differential-equation simulations.
Approach
A benchmark of three physics simulation tasks rendered as image pairs (urban sound propagation, Brown-Conrady lens distortion, bouncing ball kinematics) with task-specific physics-based metrics, used to evaluate a range of generative image-to-image models against ground-truth simulators and runtime.
Key results
- Pix2Pix achieves the lowest LoS/NLoS MAE on the baseline sound task (1.73/1.19) and runs at 0.138 ms per sample versus about 204.7 s for simulation, a speedup of roughly 20k times in the reported analysis.
- On the bouncing ball task, Pix2Pix is the most stable model, with the lowest errors on all four criteria and 11% non-evaluable predictions, while convAE and VAE have 99% and 95% non-evaluable predictions.
- Errors grow with the order of the physical relation: lens distortion Y-error is higher when only p1 is active, and bouncing ball Y-direction errors exceed X-direction errors.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Pix2Pix instead of the physical simulation | Urban Sound Propagation, Base · Urban Sound Propagation, Base | Runtime/Sample (ms): 204700 -> 0.138 | Helped |
| Add diffraction to the sound propagation task | generative sound propagation models · Urban Sound Propagation, Diffraction | MAE NLoS: Pix2Pix: 1.19 -> 3.36 | Hurt |
| Add reflections to the sound propagation task | generative sound propagation models · Urban Sound Propagation, Reflection | MAE NLoS: Pix2Pix: 1.19 -> 4.79 | Hurt |
| Activate p1 while setting p2 to zero | lens distortion models · Lens Distortion, p1 ≠ 0, p2 = 0 | Y Error: Pix2Pix: 1.02 -> 0.99 | Mixed |
| Activate p2 while setting p1 to zero | lens distortion models · Lens Distortion, p1 = 0, p2 ≠ 0 | X Error: Pix2Pix: 2.00 -> 1.77 | Mixed |
| Increase the time interval between input and prediction images | generative models for bouncing ball motion · Bouncing ball | Position X error | Hurt |
| Model higher-order physical relations | generative models across PhysicsGen tasks · PhysicsGen physical simulation tasks | prediction error | Hurt |
Surprising findings
- Stable Diffusion with cross-attention performs much worse on the bouncing ball task (position X error 40.0 px) than plain Stable Diffusion, and generates more non-evaluable images.
Limitations
- Runtime comparison is unfair because simulators were not optimized for GPU execution.
- Only three tasks and a small subset of possible physical applications are covered, so results may not transfer.
- Models fail at higher-order physical relations, and it is unclear whether this is a theoretical limit or lack of training data.
Experimental setup
Datasets: CelebA OpenStreetMap via Overpass API NoiseModelling simulation outputs Pymunk bouncing ball simulations
Benchmarks: PhysicsGen
Metrics: MAE wMAPE runtime per sample landmark X/Y error ball position/rotation/roundness error non-evaluable prediction rate
Baselines: convAE VAE U-Net Pix2Pix DDPM Stable Diffusion (with cross-attention) Stable Diffusion DDBM
Builds on: Pix2Pix DDPM DDBM Stable Diffusion U-Net
Closest papers by meaning
- PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation (NeurIPS 2025 · similarity 0.87)
- Generating Physical Dynamics under Priors (ICLR 2025 · similarity 0.86)
- PhysX-3D: Physical-Grounded 3D Asset Generation (NeurIPS 2025 · similarity 0.86)
- How Far Is Video Generation from World Model: A Physical Law Perspective (ICML 2025 · similarity 0.86)
- Compositional Scene Understanding through Inverse Generative Modeling (ICML 2025 · similarity 0.85)
- DreamPhysics: Learning Physics-Based 3D Dynamics with Video Diffusion Priors (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.