Research map / Continual learning / Class-incremental
Rethinking Query-based Transformer for Continual Image Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SimCIS, a query-based transformer baseline for class-incremental image segmentation that pre-aligns queries to image features, enforces cross-stage selection consistency, and replays "virtual queries" instead of images.
Problem
Continual image segmentation suffers from loss of plasticity and heavy dependence on input data order when query-based transformers decouple mask generation from continual learning, since their built-in objectness diminishes across stages.
Approach
Select the most semantically salient image feature locations to initialize object queries (lazy query pre-alignment), keep selections consistent across stages with a KL-based consistent selection loss, and replay stored class-specific query features (virtual queries) to mitigate forgetting in class prediction, all while training the full Mask2Former without freezing.
Key results
- On ADE20K 100-5 continual panoptic segmentation, SimCIS reaches 35.4 PQ over all classes vs. 30.8 for BalConpas-style baseline and 32.9 for ECLIPSE (per Table 1 comparison with ECLIPSE at 32.9).
- On ADE20K 100-50 continual panoptic segmentation, SimCIS achieves 40.0 PQ on all classes, nearly matching joint training (40.4).
- Virtual query replay reaches 35.4 PQ all-class with 5.9MB memory, outperforming image replay (best 34.3 at 21.9MB) while using about 27% of the storage.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace scratch-learned queries with lazy query pre-alignment (ablation) | SimCIS with pseudo labels · ADE20K 100-5 | mIoU, all classes: 13.2 -> 30.5 | Helped |
| Replace scratch-learned queries with lazy query pre-alignment (ablation) | SimCIS with pseudo labels · ADE20K 100-5 | PQ, all classes: 28.2 -> 27.9 | Hurt |
| Add consistent selection loss to QPA (ablation) | SimCIS · ADE20K 100-5 | PQ, all classes: 27.9 -> 31.8 | Helped |
| Add consistent selection loss to QPA (ablation) | SimCIS · ADE20K 100-5 | mIoU, all classes: 30.5 -> 34.5 | Helped |
| Add virtual query replay to QPA (ablation) | SimCIS · ADE20K 100-5 | mIoU, all classes: 30.5 -> 34.8 | Helped |
| Use QPA, CSL, and VQ together (ablation) | SimCIS · ADE20K 100-5 | PQ, all classes: 31.8 -> 35.4 | Helped |
| Use QPA, CSL, and VQ together (ablation) | SimCIS · ADE20K 100-5 | mIoU, all classes: 34.5 -> 38.7 | Helped |
| Increase image replay from 0 to 600 samples (ablation) | Continual panoptic segmentation · ADE20K 100-5 | PQ, all classes: 31.8 -> 34.3 | Helped |
| Use 80 virtual queries rather than no replay queries (ablation) | Continual panoptic segmentation · ADE20K 100-5 | PQ, all classes: 31.8 -> 35.4 | Helped |
| Increase virtual query replay from 80 to 160 queries (ablation) | Continual panoptic segmentation · ADE20K 100-5 | PQ, all classes: 35.4 -> 34.2 | Hurt |
| Use SimCIS instead of BalConpas | Continual panoptic segmentation · ADE20K 100-5 | PQ, all classes: 30.8 -> 35.4 | Helped |
| Use SimCIS instead of BalConpas | Continual panoptic segmentation · ADE20K 100-10 | PQ, all classes: 34.7 -> 38.1 | Helped |
Surprising findings
- Built-in objectness of query-based transformers diminishes as task sequences grow, and in the shortest two-task setting decoupled methods perform comparably to or slightly below the baseline.
Limitations
- Robustness to input order is evaluated on a single random-order setting (100-10) only
- Gains over strong baselines are smaller in some settings (e.g., 100-10 random-order results close to BalConpas)
Experimental setup
Datasets: ADE20K
Benchmarks: ADE20K
Metrics: Panoptic Quality (PQ) mIoU
Baselines: FT MiB PLOP SSUL CoMFormer BalConpas ECLIPSE EWF CoMasTRe
Builds on: Mask2Former CoMFormer BalConpas ECLIPSE MiB
Closest papers by meaning
- Continual Segmentation with Disentangled Objectness Learning and Class Recognition (CVPR 2024 · similarity 0.89)
- Towards Continual Universal Segmentation (CVPR 2025 · similarity 0.87)
- SimCS: Simulation for Domain Incremental Online Continual Segmentation (AAAI 2024 · similarity 0.86)
- CoMBO: Conflict Mitigation via Branched Optimization for Class Incremental Segmentation (CVPR 2025 · similarity 0.86)
- Beyond Background Shift: Rethinking Instance Replay in Continual Semantic Segmentation (CVPR 2025 · similarity 0.86)
- ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.