Research map / Continual learning / Online and replay-based
Towards Continual Universal Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CUE, a continual universal segmentation pipeline that decouples high-level semantic understanding (DETR-like, trained with weak memory and no distillation) from low-level boundary refinement (frozen SAM), achieving state-of-the-art on continual semantic and panoptic segmentation on VOC and ADE20K.
Problem
Continual image segmentation suffers from the stability-plasticity dilemma because high-level semantics and low-level boundary knowledge are entangled and forget at different rates; existing methods are also specialized to specific tasks and models.
Approach
Reformulate segmentation as an understanding-then-refinement pipeline: a DETR-like understanding module produces class-aware positional proposals and is updated with weak memory rather than knowledge distillation, while a frozen SAM refines proposals into masks, so high-level and low-level knowledge get different constraints. Also uses separate things/stuff decoders, semantic-aware query initializ
Key results
- On VOC 15-1 (6 steps), CUE-M reaches 84.0 all mIoU vs. prior SOTA around 71.9 (AMS-M), with new-class mIoU of 77.9.
- On ADE20K 100-10 panoptic, CUE reaches 45.5 all PQ vs. 33.9 for ECLIPSE, cutting the gap to joint training by about 10%.
- Computational analysis reports 63.5 training FPS at 147G FLOPs, versus 4.0 FPS for CoMFormer, using 20 epochs versus 128.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CUE-M instead of AMS-M | PASCAL VOC 2012 continual semantic segmentation · PASCAL VOC 2012 15-1, all classes | mIoU: 71.9 -> 84.0 | Helped |
| Use CUE-M instead of AMS-M | ADE20K continual semantic segmentation · ADE20K 100-10, all classes | mIoU: 37.6 -> 54.2 | Helped |
| Use CUE instead of ECLIPSE | ADE20K continual panoptic segmentation · ADE20K 100-10, all classes | PQ: 33.9 -> 45.5 | Helped |
| Add memory (ablation) | CUE base model · ADE20K panoptic 100-10, all classes | PQ: 38.0 -> 39.2 | Helped |
| Add separate things and stuff decoders (ablation) | CUE · ADE20K panoptic 100-10, all classes | PQ: 39.2 -> 43.2 | Helped |
| Add query initialization (ablation) | CUE · ADE20K panoptic 100-10, all classes | PQ: 43.2 -> 44.3 | Helped |
| Add grouped matching (ablation) | CUE · ADE20K panoptic 100-10, all classes | PQ: 44.3 -> 45.5 | Helped |
Surprising findings
- Memory contributes only a minor gain in ablation (+1.2 all PQ on ADE20K 100-10), which the authors attribute to ADE20K already containing many old-category samples.
- Center IoU drops moderately while boundary IoU drops sharply across continual steps, showing low-level boundary knowledge forgets faster than high-level semantics.
Limitations
- Authors describe the work as a preliminary step, far from a truly universal model.
- Memory replay is used, so comparison with memory-free methods is not fully fair; some memory-based baselines use larger memory, and the authors note this.
- Language-guided segmentation results are only in the appendix and not in the main text.
Experimental setup
Datasets: PASCAL VOC 2012 ADE20K COCO RefCOCOg
Benchmarks: PASCAL VOC 2012 ADE20K COCO
Metrics: mIoU PQ AP performance gap to joint upper bound
Baselines: MiB PLOP RCIL SSUL-M ALIFE-M AMS-M CoMFormer ECLIPSE RECALL-M
Builds on: Mask2Former MaskDINO DETR SAM DINOv2
Closest papers by meaning
- CoMFormer: Continual Learning in Semantic and Panoptic Segmentation (CVPR 2023 · similarity 0.90)
- A Continual Learning Framework for Uncertainty-Aware Interactive Image Segmentation (AAAI 2021 · similarity 0.89)
- Beyond Background Shift: Rethinking Instance Replay in Continual Semantic Segmentation (CVPR 2025 · similarity 0.88)
- Rethinking Query-based Transformer for Continual Image Segmentation (CVPR 2025 · similarity 0.87)
- ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning (CVPR 2024 · similarity 0.87)
- LENS: Learning to Segment Anything with Unified Reinforced Reasoning (AAAI 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.