Research map / Vision-language and multimodal / CLIP and zero-shot
Instruct2See: Learning to Remove Any Obstructions Across Distributions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Instruct2See, a zero-shot obstruction removal framework that treats removal as soft-hard mask restoration, using CLIP multi-modal prompts via cross-attention and a tunable mask adapter, trained on 3,984 images.
Problem
Removing arbitrary real-world obstructions (seen and unseen types such as fences, raindrops, flares, rain streaks, snow, strokes) when training data cover only a few obstruction categories and comprehensive data collection is impractical.
Approach
Unifies obstruction removal as soft-hard mask restoration: obstructions are represented by a mask from a detector plus CLIP text and visual prompts fused via cross-attention in a Restormer backbone; a tunable mask adapter refines inaccurate soft masks, and a text-similarity switch selects soft vs hard masking.
Key results
- On seen obstructions, average PSNR 29.31 and SSIM 0.9320, the best average among compared methods (Restormer second at 28.45 PSNR).
- On unseen obstructions (rain streak, snow, stroke), average PSNR 31.37 vs 29.49 for Histoformer, the second best.
- Ablation: cross-attention adds about 1.95 dB PSNR and the tunable adapter adds 0.93 dB over the fixed mask.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add a mask input to the baseline (ablation) | Instruct2See · Seen and unseen obstructions (ablation evaluation) | PSNR: 27.05 -> 28.05 | Helped |
| add cross-attention for multimodal prompts to the mask-input model (ablation) | Instruct2See · Seen and unseen obstructions (ablation evaluation) | PSNR: 28.05 -> 30.00 | Helped |
| add cross-attention for multimodal prompts to the mask-input model (ablation) | Instruct2See · Seen and unseen obstructions (ablation evaluation) | SSIM: 0.9004 -> 0.9117 | Helped |
| add the tunable mask adapter to the mask and cross-attention model (ablation) | Instruct2See · Seen and unseen obstructions (ablation evaluation) | PSNR: 30.00 -> 30.93 | Helped |
| add the tunable mask adapter to the mask and cross-attention model (ablation) | Instruct2See · Seen and unseen obstructions (ablation evaluation) | SSIM: 0.9117 -> 0.9250 | Helped |
| add a text prompt to a no-prompt model (ablation) | Instruct2See · Seen and unseen obstructions (prompt ablation) | PSNR: 28.65 -> 29.73 | Helped |
| add a text prompt to a no-prompt model (ablation) | Instruct2See · Seen and unseen obstructions (prompt ablation) | SSIM: 0.9063 -> 0.9168 | Helped |
| add a visual prompt to a no-prompt model (ablation) | Instruct2See · Seen and unseen obstructions (prompt ablation) | PSNR: 28.65 -> 30.25 | Helped |
| add a visual prompt to a no-prompt model (ablation) | Instruct2See · Seen and unseen obstructions (prompt ablation) | SSIM: 0.9063 -> 0.9215 | Helped |
| combine text and visual prompts instead of using visual prompt only (ablation) | Instruct2See · Seen and unseen obstructions (prompt ablation) | PSNR: 30.25 -> 30.93 | Helped |
| combine text and visual prompts instead of using visual prompt only (ablation) | Instruct2See · Seen and unseen obstructions (prompt ablation) | SSIM: 0.9215 -> 0.9250 | Helped |
| use ground-truth masks instead of detected masks (ablation) | Instruct2See · Seen obstructions (average) | PSNR: 29.31 -> 30.49 | Helped |
Surprising findings
- Inpainting (LaMa, RePaint) and editing (DiffEdit) methods perform poorly in zero-shot obstruction removal despite receiving obstacle masks.
- Using ground-truth masks does not uniformly help all baselines, e.g., PromptIR and WGWSNet show mixed gains on seen tasks.
Limitations
- Targets small, numerous obstacles; not intended for large obstructions where inpainting may work better.
- Sensitive to incorrect text descriptions (e.g., opaque vs semi-transparent labels) leading to suboptimal results.
- Highest parameter count among compared methods, with computational cost left for future work.
Experimental setup
Datasets: BSD UCID Flickr24K Flare7K VRDS rain streak dataset snowy dataset stroke dataset
Benchmarks: fence test set flare test set raindrop test set rain streak snow stroke
Metrics: PSNR SSIM CLIP Score LPIPS
Baselines: Restormer TransWeather PromptIR WGWSNet Histoformer XRestormer LaMa RePaint DiffEdit
Builds on: Restormer CLIP SAM2 LaMa
Closest papers by meaning
- SmartEraser: Remove Anything from Images using Masked-Region Guidance (CVPR 2025 · similarity 0.86)
- Amodal Scene Analysis via Holistic Occlusion Relation Inference and Generative Mask Completion (AAAI 2024 · similarity 0.84)
- FIRM: Flexible Interactive Reflection ReMoval (AAAI 2025 · similarity 0.84)
- Generative Omnimatte: Learning to Decompose Video into Layers (CVPR 2025 · similarity 0.84)
- Segment Anything, Even Occluded (CVPR 2025 · similarity 0.84)
- In-N-Out: Lifting 2D Diffusion Prior for 3D Object Removal via Tuning-Free Latents Alignment (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.