Research map / Image generation / Autoregressive and latent generation
Segment Any-Quality Images with Generative Latent Space Enhancement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GleSAM, which applies single-step latent diffusion denoising (with a pre-trained SD 2.1 U-Net plus LoRA) in SAM's latent space to restore high-quality features, enabling robust segmentation across image qualities, and builds the LQSeg dataset.
Problem
Segment Anything Models lose substantial accuracy on severely degraded low-quality images (noise, blur, compression, combined degradations), limiting real-world use.
Approach
Treats low-quality SAM latent features as noisy versions of high-quality features and applies a single-step denoising by a pre-trained latent diffusion U-Net (adapted with LoRA) to reconstruct HQ representations, with Feature Distribution Alignment (scaling) and Channel Replicate and Expansion to bridge the LDM and SAM latent spaces.
Key results
- On the unseen ECSSD dataset at LQ-3 degradation, GleSAM reaches 0.7332 IoU versus 0.5219 for SAM and 0.6535 for RobustSAM.
- GleSAM2 reaches 0.8611 IoU on ECSSD under RobustSeg-style degradations versus 0.8211 for SAM2.
- Training takes about 30 hours on four A100 GPUs, with 47 MB of learnable parameters versus SAM's 1250 MB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply generative latent enhancement to SAM | SAM · ThinObject-5K LQ-3 | IoU: 0.7015 -> 0.7594 | Helped |
| apply generative latent enhancement to SAM2 | SAM2 · ThinObject-5K LQ-3 | IoU: 0.7348 -> 0.7947 | Helped |
| apply generative latent enhancement to SAM | SAM · LVIS LQ-3 | IoU: 0.5316 -> 0.5535 | Helped |
| apply generative latent enhancement to SAM | SAM · ECSSD LQ-3 | IoU: 0.6675 -> 0.7332 | Helped |
| apply generative latent enhancement to SAM2 | SAM2 · COCO LQ-3 | IoU: 0.5680 -> 0.5724 | Helped |
| add Feature Distribution Alignment (ablation) | GleSAM + Gle + CRE · ECSSD and COCO unseen datasets | IoU: 0.6567 -> 0.7104 | Helped |
| fine-tune the SAM decoder with latent enhancement (ablation) | GleSAM · ECSSD and COCO unseen datasets | IoU: 0.6135 -> 0.6848 | Helped |
| replace CRE with additional encoder and decoder (ablation) | GleSAM channel expansion · Channel expansion analysis | IoU: 0.6567 -> 0.4544 | Hurt |
| replace CRE with newly trained head and tail layers (ablation) | GleSAM channel expansion · Channel expansion analysis | IoU: 0.6567 -> 0.6014 | Hurt |
| train jointly on all three degradation levels instead of one level (ablation) | GleSAM · ECSSD and COCO LQ and RobustSeg-style degradations | IoU: 0.7030 -> 0.7042 | Helped |
| increase U-Net LoRA rank from 4 to 8 (ablation) | GleSAM · LQ images | IoU: 0.7760 -> 0.7844 | Helped |
| increase U-Net LoRA rank from 8 to 16 (ablation) | GleSAM · LQ images | IoU: 0.7844 -> 0.7697 | Hurt |
Surprising findings
- Directly fine-tuning SAM's decoder or mask token on degraded images drops clear-image IoU by nearly 20 points, while GleSAM preserves clear-image performance.
- Adding new head/tail layers or an additional encoder-decoder for channel mismatch performed worse than replicating pre-trained head/tail weights.
Limitations
- Inference is slightly slower than SAM (0.38 s vs 0.32 s per image).
- Degradations are synthesized rather than drawn from real-world low-quality captures.
Experimental setup
Datasets: LVIS ThinObject-5K MSRA10K ECSSD COCO
Benchmarks: ThinObject-5K LVIS ECSSD COCO RobustSeg-style degradations
Metrics: IoU Dice Pixel Accuracy
Baselines: SAM SAM2 RobustSAM PromptIR-SAM DiffBIR-SAM
Builds on: Segment Anything Model Latent Diffusion Models LoRA RobustSAM
Closest papers by meaning
- Uncertainty-aware Fine-tuning of Segmentation Foundation Models (NeurIPS 2024 · similarity 0.87)
- Improving the Generalization of Segmentation Foundation Model under Distribution Shift via Weakly Supervised Adaptation (CVPR 2024 · similarity 0.86)
- SAM-PARSER: Fine-Tuning SAM Efficiently by Parameter Space Reconstruction (AAAI 2024 · similarity 0.86)
- Simpler Diffusion: 1.5 FID on ImageNet512 with Pixel-space Diffusion (CVPR 2025 · similarity 0.86)
- RobustSAM: Segment Anything Robustly on Degraded Images (CVPR 2024 · similarity 0.86)
- Rethinking Interactive Image Segmentation with Low Latency High Quality and Diverse Prompts (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.