Research map / Vision-language and multimodal / Cross-modal learning
LDS2AE: Local Diffusion Shared-Specific Autoencoder for Multimodal Remote Sensing Image Classification with Arbitrary Missing Modalities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LDS2AE, a local diffusion shared-specific autoencoder with a modality-shared ViT encoder and modality-specific decoders, to classify multimodal remote sensing images under arbitrary missing modalities using a single model.
Problem
Multimodal remote sensing image classification when any subset of modalities (singular or multiple, random) is missing at test time, where prior methods train a separate model per missing-modality configuration.
Approach
A modality-shared ViT encoder maps all modalities into a shared latent space, while modality-specific lightweight decoders reconstruct each modality (self and cross-modal) inside a diffusion-style denoising objective applied only to visible patches under a high masking ratio (local diffusion), which cuts training cost; fine-tuning only the classification head handles arbitrary missing inputs.
Key results
- On Trento, LDS2AE with missing LiDAR at test reaches 99.07 OA...
- On Berlin with HS and SAR training, LDS2AE outperforms Sal2RN by 3.18% OA with both modalities; ablation on Trento shows cross-modal reconstruction improves OA by 0.93-1.99%
- On Houston, missing HS or MS or LiDAR reduces OA by only 0.85%, 0.54% and 0.25% respectively relative to full input (96.99/97.47 range)
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add cross-modal reconstruction (ablation) | LDS2AE · Trento, HS and LiDAR | OA (%): 98.60 -> 99.53 | Helped |
| add cross-modal reconstruction (ablation) | LDS2AE · Trento, HS only | OA (%): 97.05 -> 98.33 | Helped |
| add cross-modal reconstruction (ablation) | LDS2AE · Trento, LiDAR only | OA (%): 89.84 -> 91.83 | Helped |
| add cross-modal reconstruction (ablation) | LDS2AE · Berlin, HS and SAR | OA (%): 72.95 -> 76.83 | Helped |
| add cross-modal reconstruction (ablation) | LDS2AE · Berlin, HS only | OA (%): 72.42 -> 73.92 | Helped |
| add cross-modal reconstruction (ablation) | LDS2AE · Berlin, SAR only | OA (%): 49.32 -> 53.21 | Helped |
| use diffusion instead of masked processing only (ablation) | LDS2AE · Trento, HS and LiDAR | OA (%): 98.65 -> 99.53 | Helped |
| use diffusion instead of masked processing only (ablation) | LDS2AE · Trento, HS only | OA (%): 97.01 -> 98.33 | Helped |
| use diffusion instead of masked processing only (ablation) | LDS2AE · Trento, LiDAR only | OA (%): 90.61 -> 91.83 | Helped |
| use diffusion instead of masked processing only (ablation) | LDS2AE · Berlin, HS and SAR | OA (%): 73.96 -> 76.83 | Helped |
| use diffusion instead of masked processing only (ablation) | LDS2AE · Berlin, HS only | OA (%): 72.51 -> 73.92 | Helped |
| use diffusion instead of masked processing only (ablation) | LDS2AE · Berlin, SAR only | OA (%): 50.17 -> 53.21 | Helped |
Limitations
- Evaluated only on three remote sensing datasets
- Fine-tuning relies on a fixed set of training samples per class (40 per class)
Experimental setup
Datasets: Trento Houston Berlin
Benchmarks: Trento Houston Berlin
Metrics: overall accuracy average accuracy kappa coefficient
Baselines: TBCNN DMAE Sal2RN HRWN MFT GLT-Net MDL-RS
Builds on: DDPM MAE SimMIM
Closest papers by meaning
- SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery (CVPR 2024 · similarity 0.85)
- DiffCLIP: Few-shot Language-driven Multimodal Classifier (AAAI 2025 · similarity 0.85)
- MDFL: Multi-Domain Diffusion-Driven Feature Learning (AAAI 2024 · similarity 0.84)
- MMM-RS: A Multi-modal, Multi-GSD, Multi-scene Remote Sensing Dataset and Benchmark for Text-to-Image Generation (NeurIPS 2024 · similarity 0.84)
- RobSense: A Robust Multi-modal Foundation Model for Remote Sensing with Static, Temporal, and Incomplete Data Adaptability (CVPR 2025 · similarity 0.84)
- Galileo: Learning Global & Local Features of Many Remote Sensing Modalities (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.