Research map / Image generation / Fast diffusion sampling
DiffusionSat: A Generative Foundation Model for Satellite Imagery
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds DiffusionSat, a latent diffusion foundation model for satellite imagery conditioned on numerical metadata (location, GSD, time) and captions, extended with a 3D ControlNet for super-resolution, temporal generation, and inpainting.
Problem
Existing diffusion models are trained on web images and cannot handle satellite data (multispectral, irregular temporal sampling, sparse captions) or the spatio-temporal inverse problems central to remote sensing.
Approach
Encode numerical satellite metadata (lat, lon, GSD, cloud cover, date) with sinusoidal embeddings and per-field MLPs added to the timestep embedding, initialize from SD 2.1, and add a 3D ControlNet with temporal attention and zero-initialized convolutions that conditions on image sequences with per-image metadata.
Key results
- Single-image 512x512 generation on fMoW validation reaches FID 15.80, versus 24.23 for SD 2.1 finetuned with metadata in captions and 117.74 for pretrained SD 2.1.
- On fMoW super-resolution from multispectral Sentinel-2 input, achieves the best LPIPS (0.6221) though DBPN has higher PSNR (11.86 vs 10.39).
- On fMoW-temporal prediction, the 3D ControlNet variant reaches SSIM 0.3983 and LPIPS 0.4304 for future prediction, beating STSR (EAD) and MCVD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use numerical metadata conditioning instead of putting metadata in the text caption | DiffusionSat single-image generation · fMoW validation, 512×512 | FID: 24.23 -> 15.80 | Helped |
| Use numerical metadata conditioning instead of putting metadata in the text caption | DiffusionSat single-image generation · fMoW validation, 512×512 | CLIP: 18.62 -> 17.20 | Hurt |
| Fine-tune SD 2.1 on the satellite dataset with captions, without metadata | SD 2.1 single-image generation · fMoW validation, 512×512 | FID: 117.74 -> 37.99 | Helped |
| Add metadata to the text caption when fine-tuning SD 2.1 | SD 2.1 single-image generation · fMoW validation, 512×512 | FID: 37.99 -> 24.23 | Helped |
| Use DiffusionSat pretraining rather than fine-tuning directly on SD weights | Texas Housing superresolution · Texas Housing, t′ > t | LPIPS: 0.4166 -> 0.3247 | Helped |
| Use DiffusionSat pretraining rather than fine-tuning directly on SD weights | Texas Housing superresolution · Texas Housing, t′ < t | LPIPS: 0.4351 -> 0.3534 | Helped |
| Use a 3D ControlNet instead of stacking images in channels with a 2D ControlNet (ablation) | DiffusionSat conditional generation · Texas Housing, t′ > t | LPIPS: 0.3874 -> 0.3247 | Helped |
| Use a 3D ControlNet instead of stacking images in channels with a 2D ControlNet (ablation) | DiffusionSat conditional generation · Texas Housing, t′ < t | LPIPS: 0.3652 -> 0.3534 | Helped |
| Use a 3D ControlNet instead of a regular ControlNet (ablation) | DiffusionSat temporal generation · fMoW-temporal, t′ > t | LPIPS: 0.5062 -> 0.4304 | Helped |
| Use a 3D ControlNet instead of a regular ControlNet (ablation) | DiffusionSat temporal generation · fMoW-temporal, t′ < t | LPIPS: 0.5307 -> 0.3937 | Helped |
| Use DiffusionSat pretraining with 3D ControlNet rather than SD with 3D ControlNet | Temporal generation · fMoW-temporal, t′ > t | LPIPS: 0.5523 -> 0.4304 | Helped |
Surprising findings
- Higher PSNR/SSIM baselines such as DBPN are reported to favor blurriness over perceptual quality, so LPIPS is emphasized instead.
Limitations
- Super-resolution outputs can hallucinate details and show variance across samples.
- Geographic bias evaluation is limited to FID/LPIPS by location and cannot capture other biases; data is concentrated in developed regions.
Experimental setup
Datasets: fMoW Satlas SpaceNet Texas housing (STSR dataset) fMoW-Sentinel xBD
Benchmarks: fMoW Texas housing fMoW-temporal xBD
Metrics: FID IS CLIP score SSIM PSNR LPIPS MSE
Baselines: Stable Diffusion 2.1 Pix2Pix DBPN SRGAN STSR MCVD
Builds on: Stable Diffusion ControlNet VideoLDM LDM
Closest papers by meaning
- MMM-RS: A Multi-modal, Multi-GSD, Multi-scene Remote Sensing Dataset and Benchmark for Text-to-Image Generation (NeurIPS 2024 · similarity 0.90)
- EcoMapper: Generative Modeling for Climate-Aware Satellite Imagery (ICML 2025 · similarity 0.89)
- SatSynth: Augmenting Image-Mask Pairs through Diffusion Models for Aerial Semantic Segmentation (CVPR 2024 · similarity 0.89)
- GeoCore-9B: Towards Geo-Aware Generative Foundation Models in Earth Observation (NeurIPS 2026 · similarity 0.87)
- Satellite to GroundScape - Large-scale Consistent Ground View Generation from Satellite Views (CVPR 2025 · similarity 0.86)
- Building Bridges across Spatial and Temporal Resolutions: Reference-Based Super-Resolution via Change Priors and Conditional Diffusion Model (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.