Research map / Label-efficient and robust learning / Domain adaptation and generalization
Semantic Library Adaptation: LoRA Retrieval and Fusion for Open-Vocabulary Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SemLA, a training-free test-time adaptation method for open-vocabulary semantic segmentation that retrieves and fuses LoRA adapters from a library indexed by CLIP domain centroids.
Problem
Open-vocabulary semantic segmentation models degrade under large train-test domain shifts, and existing domain adaptation methods are slow, need source data, or were not applied to OV segmentation.
Approach
Index a library of LoRA adapters (one per training domain) by CLIP centroids of their training images; at test time, retrieve the top-K nearest adapters to each input image and fuse them by distance-based softmax weights via LoRA concatenation, producing a per-image adapted model without training.
Key results
- SemLA reaches h-mean 54.16 on the leave-one-out 20-domain CAT-Seg benchmark vs 39.39 zero-shot and 51.89 uniform LoRA merging.
- With SED as backbone, SemLA improves h-mean from 35.56 (zero-shot) to 50.57 (uniform merging 48.60).
- In the all-inclusive setting with tau=0.005, SemLA reaches h-mean 60.57, close to the Oracle's 61.05.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Proximity-weighted CLIP-based LoRA adapter selection and merging (SemLA) instead of uniform merging | CAT-Seg · 20-domain leave-one-out benchmark | h-mean mIoU: +2.27 over uniform merging (54.16 vs 51.89) | Helped |
| Proximity-weighted LoRA merging (SemLA) instead of zero-shot CAT-Seg | CAT-Seg · 20-domain leave-one-out benchmark | h-mean mIoU: +14.77 (54.16 vs 39.39) | Helped |
| Proximity-weighted LoRA merging (SemLA) instead of uniform merging | SED · 20-domain leave-one-out benchmark | h-mean mIoU: +1.97 (50.57 vs 48.60) | Helped |
| Output-level late fusion (uniform or proximity-weighted) instead of parameter-level LoRA merging (ablation) | CAT-Seg · 20-domain leave-one-out benchmark | h-mean mIoU: -1.99 for SemLA late fusion vs SemLA (52.09 vs 54.16) | Hurt |
| Low softmax temperature (tau=0.005) so target-domain adapter dominates, with target adapter in library (ablation) | SemLA on CAT-Seg · All-inclusive library | h-mean mIoU: 60.57 vs Oracle 61.05 | No clear effect |
| Higher temperature (tau=0.05) promoting broader LoRA fusion instead of near-hard selection, all-inclusive library (ablation) | SemLA on CAT-Seg · All-inclusive library | h-mean mIoU: -1.78 vs tau=0.005 (58.79 vs 60.57) | Hurt |
| Increasing number of adapters with uniform averaging versus proximity-based selection | CAT-Seg · 20-domain leave-one-out benchmark | h-mean mIoU | No clear effect |
Surprising findings
- Late-fusion variants (output-level averaging or weighting) underperform parameter-level LoRA merging despite higher computational cost.
- When all target-domain adapters are in the library, SemLA with higher temperature (more fusion) overcomes the Oracle on 14 of 20 domains.
Limitations
- No explicit vocabulary alignment between source and target label spaces.
- Scaling to thousands of adapters raises challenges in identifying library gaps and weaknesses.
- Evaluated only on segmentation; multi-task settings such as panoptic segmentation or depth estimation not tested.
Experimental setup
Datasets: Cityscapes BDD100K Mapillary Vistas Indian Driving Dataset ACDC MUSES ADE20K PC59 NYU COCONutL
Benchmarks: 20-domain OV domain adaptation benchmark built over 10 datasets (ACDC, MUSES, Cityscapes, BDD100K, MV, ADE150, IDD, PC59
Metrics: mIoU harmonic mean mIoU
Baselines: Zero-shot CAT-Seg Uniform LoRA Merging Uniform (Late Fusion) SemLA (Late Fusion) Oracle single-domain adapters
Builds on: CAT-Seg LoRA CLIP Model soups / uniform LoRA merging (Li et al.) LoraRetriever
Closest papers by meaning
- Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation (NeurIPS 2025 · similarity 0.90)
- SemiDAViL: Semi-supervised Domain Adaptation with Vision-Language Guidance for Semantic Segmentation (CVPR 2025 · similarity 0.89)
- CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.89)
- Universal Domain Adaptation for Semantic Segmentation (CVPR 2025 · similarity 0.89)
- SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.88)
- Training-free Open-Vocabulary Semantic Segmentation via Diverse Prototype Construction and Sub-region Matching (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.