Research map / Continual learning / Continual fine-tuning of LLMs
OVS Meets Continual Learning: Towards Sustainable Open-Vocabulary Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies open-vocabulary segmentation under sequentially arriving datasets and proposes ConOVS, an MoE-style continual learning method that interpolates per-dataset decoder task vectors using MVN-based likelihood estimates of input proximity.
Problem
Open-vocabulary segmentation models are trained once on a fixed dataset, so they degrade on unseen classes; when new datasets arrive sequentially, retraining is costly, fine-tuning causes catastrophic forgetting, and existing continual learning methods assume a finite class set.
Approach
Fine-tune only the decoder per incremental dataset to create expert task vectors, estimate per-dataset interpolation factors from multivariate normal likelihoods of image and text embeddings (softmax, then element-wise max), and build a per-sample decoder as the base decoder plus weighted task vectors.
Key results
- In scenario S1 with fc-clip, ConOVS gains +0.3 PQ on COCO pre-training, +20.2 on Cityscapes incremental, and +2.5 on ADE20K zero-shot, while fine-tuning drops -22.7 on COCO.
- In scenario S3 (Cityscapes then ADE20K), ConOVS reaches 47.0 PQ on ADE20K and 64.3 on Cityscapes, outperforming fine-tuning and retraining on the incremental datasets.
- ConOVS achieves the best score on all eight zero-shot datasets in S3 (e.g., LVIS 23.1 mAP vs 21.5 for retraining and 20.5 for fc-clip).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ConOVS instead of fc-clip without incremental training | fc-clip on Cityscapes incremental S1 · Cityscapes | PQ: +20.2 | Helped |
| Use ConOVS instead of fc-clip without incremental training | fc-clip on ADE20K incremental S2 · ADE20K | PQ: +23.8 | Helped |
| Use ConOVS instead of fc-clip without incremental training | fc-clip on COCO pre-training S2 · COCO | PQ: +1.7 | Helped |
| Use ConOVS instead of fc-clip without incremental training | fc-clip on Cityscapes zero-shot S2 · Cityscapes | PQ: +0.9 | Helped |
| Combine image and text distributions rather than image-only estimates (ablation) | ConOVS on Cityscapes incremental S1 · Cityscapes | PQ: 43.4 -> 64.3 | Helped |
| Combine image and text distributions rather than text-only estimates (ablation) | ConOVS on Cityscapes incremental S1 · Cityscapes | PQ: 60.7 -> 64.3 | Helped |
| Replace MVN distribution with K-means clustering (ablation) | ConOVS on COCO pre-training · COCO | PQ: 50.4 -> 42.4 | Hurt |
| Replace MVN distribution with kernel density estimation (ablation) | ConOVS on Cityscapes incremental · Cityscapes | PQ: 64.3 -> 57.4 | Hurt |
| Replace softmax with argmax for interpolation factors (ablation) | ConOVS on LVIS · LVIS | mAP: 21.3 -> 23.1 | Hurt |
| Replace full decoder tuning/interpolation with prompt tuning and single-expert selection (ablation) | ConOVS on Cityscapes incremental S1 · Cityscapes | PQ: 48.9 -> 64.3 | Hurt |
| Fine-tune only the encoder last block rather than only the decoder (ablation) | ConOVS on Cityscapes incremental S1 · Cityscapes | PQ: 50.4 -> 64.4 | Hurt |
| Fine-tune encoder LayerNorm rather than only the decoder (ablation) | ConOVS on Cityscapes incremental S1 · Cityscapes | PQ: 44.0 -> 64.4 | Hurt |
Surprising findings
- Retraining from scratch on the joint data underperforms ConOVS on incremental and zero-shot datasets in S3, which the authors attribute to needing more iterations to converge.
- Continual learning methods adapted to OVS (ER, LwF, EWC, ECLIPSE) often perform worse than simple baselines on the pre-training dataset.
Limitations
- Each input sample yields a unique decoder weight, limiting batch size to the encoder; per-sample decoding is required.
- Storage grows with the number of datasets (about 80MB decoder per dataset, 6.11% of model size).
- Results reported without error bars because multiple runs were too costly.
Experimental setup
Datasets: COCO Cityscapes ADE20K BDD100K Mapillary Vistas
Benchmarks: COCO Cityscapes ADE20K LVIS BDD100K Mapillary Vistas PC-59 PC-459 PAS-20 PAS-21 A-847
Metrics: PQ mAP mIoU
Baselines: fc-clip X-Decoder Fine-tuning Retraining ER LwF EWC ECLIPSE OneFormer
Builds on: fc-clip X-Decoder task arithmetic ECLIPSE learning without forgetting
Closest papers by meaning
- Transferable and Principled Efficiency for Open-Vocabulary Segmentation (CVPR 2024 · similarity 0.90)
- Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.87)
- Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation (NeurIPS 2025 · similarity 0.87)
- Mask-Adapter: The Devil is in the Masks for Open-Vocabulary Segmentation (CVPR 2025 · similarity 0.87)
- Training-free Open-Vocabulary Semantic Segmentation via Diverse Prototype Construction and Sub-region Matching (AAAI 2025 · similarity 0.86)
- CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.