Research map / Continual learning / Class-incremental
Continual Segmentation with Disentangled Objectness Learning and Class Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoMasTRe, a two-stage query-based segmenter that decouples continual segmentation into class-agnostic objectness learning (with objectness/mask/position distillation) and class recognition (with task queries, task-specific classifiers, and multi-label class distillation).
Problem
Continual semantic segmentation suffers catastrophic forgetting of old classes and poor transfer to new classes when treated as per-pixel classification; the paper asks whether query-based mask classification can be exploited to mitigate this.
Approach
Query-based segmenters have built-in objectness that transfers to unseen classes and resists forgetting; CoMasTRe splits segmentation into stage 1 class-agnostic mask proposal/objectness learning and stage 2 class recognition, distilling objectness and class knowledge separately.
Key results
- On VOC 19-1, CoMasTRe improves new-class mIoU by 32.16 p.p over per-pixel baselines.
- On ADE20K 100-5, CoMasTRe improves all-class mIoU by 1.65 p.p over CoMFormer (38.64 avg vs 36.50 avg), and on 100-10 by 2.11 p.p.
- On VOC 15-1, CoMasTRe reaches 69.77 base-class mIoU and 63.54 all-class mIoU, beating RCIL by 4.14 p.p on all classes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CoMasTRe instead of per-pixel and query-based baselines | Continual segmentation · PASCAL VOC 15-1 | all mIoU (%): 63.54 vs. 59.40 | Helped |
| Use CoMasTRe instead of CoMFormer | Continual segmentation · ADE20K 100-10 | all mIoU (%): 34.41 vs. 32.30 | Helped |
| Pretrain stage 1 on COCO without PASCAL classes | CoMasTRe · PASCAL VOC 15-1 | all mIoU (%): 63.54 -> 64.88 | Helped |
| Add objectness-score distillation (ablation) | CoMasTRe objectness learning · PASCAL VOC 15-1 | all mIoU (%): 18.44 -> 59.64 | Helped |
| Add mask distillation (ablation) | CoMasTRe objectness learning · PASCAL VOC 15-1 | all mIoU (%): 57.61 -> 59.64 | Helped |
| Add position distillation (ablation) | CoMasTRe objectness learning · PASCAL VOC 15-1 | all mIoU (%): 59.64 -> 63.54 | Helped |
| Add unmatched-query class distillation (ablation) | CoMasTRe class recognition · PASCAL VOC 15-1 | all mIoU (%): 36.47 -> 59.12 | Helped |
| Add matched-query class distillation (ablation) | CoMasTRe class recognition · PASCAL VOC 15-1 | all mIoU (%): 59.12 -> 63.54 | Helped |
| Add task queries (ablation) | CoMasTRe class recognition · PASCAL VOC 15-1 | all mIoU (%): 51.36 -> 55.48 | Helped |
| Add auxiliary loss (ablation) | CoMasTRe class recognition · PASCAL VOC 15-1 | all mIoU (%): 51.36 -> 58.14 | Helped |
| Use focal loss instead of cross-entropy (ablation) | CoMasTRe class recognition · PASCAL VOC 15-1 | all mIoU (%): 51.36 -> 60.18 | Helped |
Surprising findings
- Objectness transfer from COCO pretraining (with PASCAL classes removed) gives a 1.34 p.p all-class gain on VOC 15-1.
- Removing the mask-distillation loss causes only about 2 p.p drop, suggesting mask proposals are forgetting-robust.
Limitations
- Evaluated only on semantic segmentation with two datasets (VOC, ADE20K); extension to panoptic/instance segmentation is left open.
- Relies on a background-excluded annotation and panoptic inference to estimate background IoU.
- Performance on ADE20K 100-50 is only competitive with CoMFormer on new classes.
Experimental setup
Datasets: PASCAL VOC 2012 ADE20K COCO 2017 (modified, PASCAL classes removed, for objectness transfer pretraining)
Benchmarks: PASCAL VOC 2012 ADE20K
Metrics: mIoU base-class mIoU incremented-class mIoU avg mIoU across steps
Baselines: MiB SDR PLOP REMINDER RCIL CoMFormer Mask2Former
Builds on: Mask2Former MaskFormer DyTox CoMFormer PLOP
Closest papers by meaning
- Rethinking Query-based Transformer for Continual Image Segmentation (CVPR 2025 · similarity 0.89)
- Beyond Background Shift: Rethinking Instance Replay in Continual Semantic Segmentation (CVPR 2025 · similarity 0.87)
- ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning (CVPR 2024 · similarity 0.85)
- CoMBO: Conflict Mitigation via Branched Optimization for Class Incremental Segmentation (CVPR 2025 · similarity 0.85)
- OVS Meets Continual Learning: Towards Sustainable Open-Vocabulary Segmentation (NeurIPS 2025 · similarity 0.85)
- Towards Continual Universal Segmentation (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.