Research map / Continual learning / Class-incremental
Beyond Background Shift: Rethinking Instance Replay in Continual Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Enhanced Instance Replay (EIR), which stores old-class instances instead of images and fuses them into new images via class combination, strategic placement, and region-specific distillation to fix background shift in continual semantic segmentation.
Problem
Continual semantic segmentation suffers from background shift and partial labeling: stored and new images only annotate current/old classes, so unannotated old and future classes are confused with background, making image replay ineffective.
Approach
Store individual old-class instances rather than whole images, dynamically combine them with new-class images using pseudo-label-based class selection, place them in background regions via a strategic placement rule with mixup, and apply a region-specific knowledge distillation loss to the fused regions.
Key results
- On Pascal VOC 2012 5-3 overlapped, EIR reaches 66.41 mIoU overall versus 57.30 for random copy-paste instance replay and 42.49 for image replay.
- With Swin-B on VOC 10-1 (11 steps), EIR achieves 73.7 mIoU versus 70.5 for CoinSeg.
- On ADE20K 100-10 with ResNet-101, EIR achieves 34.3 mIoU versus 34.5 for MicroSeg-M, so it is not uniformly best.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use enhanced instance replay instead of image replay | Continual semantic segmentation · Pascal VOC 5-3 overlapped | mIoU: 42.49 -> 66.41 | Helped |
| Use enhanced instance replay instead of the strongest listed baseline | Continual semantic segmentation · Pascal VOC 5-3 overlapped, all classes | mIoU: 64.1 -> 66.4 | Helped |
| Use enhanced instance replay instead of the strongest listed baseline | Continual semantic segmentation · Pascal VOC 15-5, all classes | mIoU: 75.2 -> 74.2 | Hurt |
| Use enhanced instance replay instead of the strongest listed baseline | Continual semantic segmentation · ADE20K 100-10, all classes | mIoU: 34.5 -> 34.3 | Hurt |
| Add dynamic old-class combination (ablation) | EIR · Pascal VOC 5-3 overlapped | mIoU: 58.52 -> 63.16 | Helped |
| Add strategic instance placement (ablation) | EIR · Pascal VOC 5-3 overlapped | mIoU: 59.99 -> 63.16 | Helped |
| Add enhanced replay training (ablation) | EIR with dynamic combination and strategic placement · Pascal VOC 5-3 overlapped | mIoU: 63.16 -> 66.41 | Helped |
| Use pseudo-labels to select instance classes instead of random selection (ablation) | EIR · Pascal VOC 5-3 overlapped | mIoU: 64.58 -> 66.41 | Helped |
| Use pseudo-labels to select instance classes instead of class-order selection (ablation) | EIR · Pascal VOC 5-3 overlapped | mIoU: 64.04 -> 66.41 | Helped |
| Add strategic instance placement instead of no placement (ablation) | EIR · Pascal VOC 5-3 overlapped | mIoU: 44.75 -> 66.41 | Helped |
| Use strategic placement instead of random paste (ablation) | EIR · Pascal VOC 5-3 overlapped | mIoU: 63.38 -> 66.41 | Helped |
| Use RSKD instead of UNKD (ablation) | EIR · Pascal VOC 5-3 overlapped | mIoU: 54.74 -> 66.41 | Helped |
Surprising findings
- Naive vanilla instance replay performs better than image replay only modestly and random copy-paste is substantially worse than the strategic method, showing placement matters.
- Random paste improves overall mIoU over no placement but degrades old-class performance.
Limitations
- Does not fully resolve background shift because instances cannot cover all background-shift regions in new images.
- On ADE20K 100-10 ResNet-101, EIR is slightly below MicroSeg-M overall.
Experimental setup
Datasets: Pascal VOC 2012 ADE20K
Benchmarks: Pascal VOC 2012 ADE20K
Metrics: mIoU
Baselines: MiB PLOP SSUL-M MicroSeg-M RCIL EWF TIKP ILT SSUL MicroSeg CoinSeg
Builds on: MiB PLOP SSUL DKD PLOPLong
Closest papers by meaning
- TIKP: Text-to-Image Knowledge Preservation for Continual Semantic Segmentation (AAAI 2024 · similarity 0.90)
- Adaptive Prototype Replay for Class Incremental Semantic Segmentation (AAAI 2025 · similarity 0.89)
- Enhancing Visual Continual Learning with Language-Guided Supervision (CVPR 2024 · similarity 0.88)
- Towards Continual Universal Segmentation (CVPR 2025 · similarity 0.88)
- Continual Segmentation with Disentangled Objectness Learning and Class Recognition (CVPR 2024 · similarity 0.87)
- FALCON: Fairness Learning via Contrastive Attention Approach to Continual Semantic Scene Understanding (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.