Research map / Vision-language and multimodal / CLIP and zero-shot
DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
DUNIA learns pixel-sized Earth observation embeddings by contrastively aligning Sentinel-1/2 images with GEDI full-waveform LiDAR and with multi-temporal image embeddings, then uses them for zero-shot retrieval and low-shot fine-tuning on seven EO tasks.
Problem
Most self-supervised multimodal EO models produce coarse patch-level embeddings, which limits pixel-level prediction and integration of LiDAR waveform data such as GEDI full-waveform measurements.
Approach
A transformer encoder with two convolutional decoders produces pixel-level embeddings aligned both with GEDI waveform embeddings (vertical structure, via Zero-CL) and with multi-temporal image embeddings (horizontal structure, via VICReg), plus reconstruction losses and a latent diffusion model that generates waveforms from pixel embeddings.
Key results
- Zero-shot DUNIA with KNN=50 gets RMSE 2.0 m (r=.93) for canopy height W_rh and 11.7 (r=.89) for canopy cover, outperforming specialist models; zero-shot CLC+ wF1 is 80.1%.
- Fine-tuned DUNIA reaches 90.3 wF1 on CLC+ Backbone, slightly above AnySat (90.1), and 1.3 m RMSE (r=.95) for canopy height at 5K images.
- Zero-shot PASTIS crop mapping is weak at 56.2 OA vs 84.2 for SOTA, attributed to single median composites missing phenology.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DUNIA pixel embeddings with KNN=50 instead of the supervised specialist | Canopy height (W_rh), zero-shot · Canopy height (W_rh) | RMSE: 5.6 -> 2.0 | Helped |
| Use DUNIA pixel embeddings with KNN=50 instead of the supervised specialist | Canopy cover (W_c), zero-shot · Canopy cover (W_c) | RMSE: 22.1 -> 11.7 | Helped |
| Use DUNIA pixel embeddings with KNN=50 instead of the supervised specialist | Plant area index (W_pai), zero-shot · Plant area index (W_pai) | RMSE: 1.5 -> 0.71 | Helped |
| Use DUNIA embeddings with KNN=50 for crop mapping | PASTIS, zero-shot · PASTIS | OA: 84.2 -> 56.2 | Hurt |
| Reduce retrieval neighbors from KNN=50 to KNN=5 | PureForest tree species zero-shot classification · PureForest | wF1: 73.8 -> 76.0 | Helped |
| Reduce retrieval neighbors from KNN=50 to KNN=5 | CLC+ and PASTIS zero-shot classification · CLC+ and PASTIS | classification score: 80.1 -> 74.3; 56.2 -> 54.1 | Hurt |
| Fine-tune DUNIA rather than use zero-shot retrieval | CLC+ land cover mapping · CLC+ | wF1: 80.1 -> 90.3 | Helped |
| Fine-tune DUNIA rather than use zero-shot retrieval | PureForest tree species classification · PureForest | wF1: 76.0 -> 82.2 | Helped |
| Use DUNIA instead of AnySat | Fine-tuned canopy height mapping · Canopy height (W_rh) | RMSE: 2.8 -> 1.3 | Helped |
| Use DUNIA instead of AnySat | Fine-tuned canopy cover mapping · Canopy cover (W_c) | RMSE: 12.1 -> 9.8 | Helped |
| Use DUNIA instead of AnySat | Fine-tuned plant area index mapping · Plant area index (W_pai) | RMSE: 0.95 -> 0.62 | Helped |
| Use DUNIA instead of AnySat | Fine-tuned crop mapping · PASTIS | wF1: 81.1 -> 77.0 | Hurt |
Surprising findings
- Zero-shot retrieval with embeddings sometimes beats supervised specialist models on vertical structure products.
- Zero-shot waveform retrieval outperforms the fine-tuned models on vertical structure products; all baseline FMs trail sharply on waveform tasks.
Limitations
- Relies on static median composites, which hurts crop type mapping that needs multi-temporal information.
- Pre-trained model is tied to the area and year of training (metropolitan France) and likely needs re-pretraining elsewhere.
- Crop and land cover zero-shot performance drops significantly with smaller retrieval databases.
Experimental setup
Datasets: Sentinel-1 Sentinel-2 GEDI PureForest CLC+ Backbone PASTIS
Benchmarks: PureForest CLC+ Backbone PASTIS GEDI vertical structure dataset
Metrics: weighted F1 overall accuracy RMSE Pearson correlation
Baselines: SatMAE DOFA DeCUR CROMA AnySat
Builds on: VICReg Zero-CL VQ-VAE latent diffusion models Lion optimizer
Closest papers by meaning
- AnySat: One Earth Observation Model for Many Resolutions, Scales, and Modalities (CVPR 2025 · similarity 0.86)
- OAM-TCD: A globally diverse dataset of high-resolution tree cover maps (NeurIPS 2024 · similarity 0.86)
- SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery (CVPR 2024 · similarity 0.85)
- InstructSAM: A Training-free Framework for Instruction-Oriented Remote Sensing Object Recognition (NeurIPS 2025 · similarity 0.85)
- FoMo: Multi-Modal, Multi-Scale and Multi-Task Remote Sensing Foundation Models for Forest Monitoring (AAAI 2025 · similarity 0.85)
- Learnable Earth Parser: Discovering 3D Prototypes in Aerial Scans (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.