Research map / Continual learning / Class-incremental
FALCON: Fairness Learning via Contrastive Attention Approach to Continual Semantic Scene Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FALCON, a continual semantic segmentation method combining a fairness-weighted contrastive clustering loss with an attention-based visual grammar model to handle catastrophic forgetting, fairness, and unknown classes.
Problem
Continual semantic scene segmentation must learn new classes without forgetting old ones, while handling background shift from unknown classes and unfair predictions between major and minor classes caused by imbalanced pixel distributions.
Approach
A fairness contrastive clustering loss adds a scaling factor and a learnable transition vector per cluster to equalize enforcement between major and minor classes, and a self-attention visual grammar model learns feature-cluster distribution structure to assign pixels to multiple unknown clusters.
Key results
- On ADE20K 100-50 with DeepLab-V3, FALCON reaches 37.9% all-class mIoU, and 36.4% on ADE20K 100-10, above prior FairCL (36.4 vs 34.7 on 100-10).
- With the Transformer backbone, FALCON outperforms CoMFormer by +3.5%, +8.0%, and +2.6% on ADE20K 100-50, 100-10, and 100-5.
- On Pascal VOC 15-1 and Cityscapes 1-1, FALCON with DeepLab-V3 improves over FairCL by up to 69.83% and +6.02% mIoU respectively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace vanilla contrastive clustering loss with fairness contrastive clustering loss (ablation) | FALCON · ADE20K 100-50 | mIoU (all): 34.8 -> 37.9 | Helped |
| Replace vanilla contrastive clustering loss with fairness contrastive clustering loss (ablation) | FALCON · ADE20K 100-10 | mIoU (all): 33.2 -> 36.4 | Helped |
| Use visual grammar instead of nearest-cluster assignment (ablation) | FALCON · ADE20K 100-50 | mIoU (all): 33.4 -> 37.9 | Helped |
| Adaptively update visual grammar rather than keep it fixed (ablation) | FALCON · ADE20K 100-50 | mIoU (all): 35.6 -> 37.9 | Helped |
| Use visual grammar instead of nearest-cluster assignment (ablation) | FALCON · ADE20K 100-10 | mIoU (all): 31.5 -> 36.4 | Helped |
| Adaptively update visual grammar rather than keep it fixed (ablation) | FALCON · ADE20K 100-10 | mIoU (all): 34.9 -> 36.4 | Helped |
| Set α to 0.05 rather than 0.1 (ablation) | FALCON · ADE20K 100-50 | mIoU (all): 35.3 -> 37.9 | Helped |
| Set α to 0.005 rather than 0.05 (ablation) | FALCON · ADE20K 100-50 | mIoU (all): 37.9 -> 34.5 | Hurt |
| Set α to 0.01 rather than 0.1 (ablation) | FALCON · Pascal VOC 15-5 | mIoU (all): 69.3 -> 73.5 | Helped |
| Set α to 0.005 rather than 0.01 (ablation) | FALCON · Pascal VOC 15-5 | mIoU (all): 73.5 -> 68.5 | Hurt |
Surprising findings
- Adding the fairness loss sharply improves minor-class mIoU on ADE20K 100-50 (from 15.2 to 24.5 on new classes for the same setup), suggesting the vanilla contrastive clustering loss strongly biases toward major classes.
Limitations
- Hyperparameters such as alpha are set per dataset and sensitive; too small alpha weakens cluster compactness.
- Cluster initialization and learning hyperparameters are chosen to support theory and may limit generality.
- Visual grammar model requires pre-training on ImageNet and updates each step, adding complexity.
Experimental setup
Datasets: ADE20K Pascal VOC Cityscapes ImageNet
Benchmarks: ADE20K 100-50 ADE20K 100-10 ADE20K 100-5 Pascal VOC 15-5 Pascal VOC 15-1 Pascal VOC 10-1 Cityscapes 11-5 Cityscapes 11-1 Cityscapes 1-1
Metrics: mIoU major-class mIoU minor-class mIoU
Baselines: MiB PLOP RCIL REMINDER FairCL CoMFormer ECLIPSE CoMasTRe SSUL LWF-MC ILT
Builds on: contrastive clustering parametric contrastive learning DeepLab-V3 SegFormer Visual grammar (Lee et al. clustering with self-attention)
Closest papers by meaning
- Beyond Background Shift: Rethinking Instance Replay in Continual Semantic Segmentation (CVPR 2025 · similarity 0.87)
- Towards Continual Universal Segmentation (CVPR 2025 · similarity 0.85)
- Enhancing Visual Continual Learning with Language-Guided Supervision (CVPR 2024 · similarity 0.85)
- TIKP: Text-to-Image Knowledge Preservation for Continual Semantic Segmentation (AAAI 2024 · similarity 0.85)
- Adaptive Visual Scene Understanding: Incremental Scene Graph Generation (NeurIPS 2024 · similarity 0.85)
- ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.