Research map / Vision-language and multimodal / CLIP and zero-shot
Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MLMP, a plug-and-play fully test-time adaptation method for open-vocabulary semantic segmentation with VLMs that combines uncertainty-aware multi-level ViT feature fusion and multi-prompt entropy minimization at local and CLS levels, plus a 87-scenario OVSS TTA benchmark.
Problem
Test-time adaptation of vision-language models for open-vocabulary semantic segmentation, which is vulnerable to corruptions and domain shifts and has been overlooked in TTA research that focuses on classification.
Approach
Adapts ViT LayerNorm parameters by minimizing entropy over patch tokens fused from intermediate layers with entropy-based confidence weights, and over multiple prompt templates at the loss level, plus an image-level entropy term on the CLS token.
Key results
- On V20-C (15 corruptions averaged) with NACLIP ViT-L/14, MLMP reaches 77.58 mIoU vs 69.01 no-adaptation and 69.33 for TENT.
- MLMP improves clean V20 from 75.91 to 83.76 mIoU and gains +3.86 mIoU on Cityscapes.
- On ACDC, MLMP averages 30.28 mIoU vs 23.35 no adaptation and 26.29 TENT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply MLMP instead of no adaptation | NACLIP ViT-L/14 · V20 (Original) | mIoU: +7.85 | Helped |
| apply MLMP instead of no adaptation | NACLIP ViT-L/14 · V20-C Average | mIoU: +8.60 | Helped |
| apply MLMP instead of no adaptation | NACLIP ViT-L/14 · ACDC | mIoU: +6 | Helped |
| apply MLMP instead of no adaptation | NACLIP ViT-L/14 · GTA-V | mIoU: +3.8 | Helped |
| use layers L7–24 instead of only L24 (ablation) | MLMP · V20 (Original) | mIoU: 77.00 -> 81.67 | Helped |
| use layers L7–24 instead of only L24 (ablation) | MLMP · V20-C Average | mIoU: 69.33 -> 74.90 | Helped |
| use all layers L1–24 instead of L7–24 (ablation) | MLMP · V20 (Original) | mIoU: 81.67 -> 78.79 | Hurt |
| use all layers L1–24 instead of L7–24 (ablation) | MLMP · V20-C Average | mIoU: 74.90 -> 72.02 | Hurt |
| add uncertainty-aware weighting to multi-level fusion (ablation) | MLMP · V20 | mIoU: +4.29% | Helped |
| add uncertainty-aware weighting to multi-level fusion (ablation) | MLMP · V20-C | mIoU: +3.31% | Helped |
| add image-level entropy term (ablation) | MLMP · V20 | mIoU: +1.74% | Helped |
| add image-level entropy term (ablation) | MLMP · V20-C | mIoU: +2.66% | Helped |
Surprising findings
- Baselines adapted from classification such as WATT and CLIPArTT often degrade segmentation performance below the no-adaptation baseline.
- TENT yields a performance drop on corrupted V20-C with a single test sample while MLMP improves consistently.
Limitations
- Layer weighting relies on a shared projection head, which may not reflect each layer's characteristics (stated by the authors).
- Corruptions are synthetic at severity 5 and many benchmark scenarios are derived from them, so real-shift coverage is limited to ACDC and GTA-V.
Experimental setup
Datasets: Pascal VOC 20 Pascal VOC 21 Pascal Context 59 Pascal Context 60 Cityscapes COCO-Stuff COCO-Object ACDC GTA-V ImageNet-C-style corruptions
Benchmarks: Pascal VOC 20 Pascal VOC 21 Pascal Context 59 Pascal Context 60 Cityscapes COCO-Stuff COCO-Object ACDC GTA-V
Metrics: mIoU
Baselines: No Adapt. TENT TPT WATT CLIPArTT
Builds on: TENT NACLIP CLIP WATT CLIPArTT
Closest papers by meaning
- Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.92)
- The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models (NeurIPS 2025 · similarity 0.91)
- Frustratingly Easy Test-Time Adaptation of Vision-Language Models (NeurIPS 2024 · similarity 0.91)
- What Drives Test-Time Adaptation for CLIP? A Controlled Empirical Study from an Update Perspective (NeurIPS 2026 · similarity 0.90)
- Mask-Adapter: The Devil is in the Masks for Open-Vocabulary Segmentation (CVPR 2025 · similarity 0.90)
- BoostAdapter: Improving Vision-Language Test-Time Adaptation via Regional Bootstrapping (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.