Research map / Label-efficient and robust learning / Recognition and long-tail

Beyond Prototypes: Semantic Anchor Regularization for Better Representation Learning

AAAI 2024 · Label-efficient and robust learning › Recognition and long-tail · approach: Contrastive learning for imbalanced data · 10 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Semantic Anchor Regularization (SAR), which pulls pixel/feature representations toward pre-defined class anchors that are projected into semantic space, classified by a shared classifier, and maintained via EMA, improving segmentation and classification plug-and-play.

Problem

Prototype-based representation learning via Expectation-Maximization couples class centers to learned features and training-data distribution, introducing biases that hurt tail classes in long-tailed training data (e.g., semantic segmentation).

Approach

Instead of deriving class prototypes from learned features, use pre-defined class anchors as fixed feature centroids; project them via a trainable embedding layer, enforce their inter-class separability with a classifier-aware auxiliary cross-entropy loss (reweighted), smooth them with EMA, and unidirectionally pull pixel features toward them.

Key results

• On Cityscapes, HRNet+SAR reaches 81.4 mIoU vs 79.9 for HRNet (+1.5), and outperforms ProtoSeg (81.1) under the fair comparison.
• On ADE20K, HRNet+SAR improves mIoU from 42.0 to 42.8 (+0.8) and SegFormer+SAR gains +0.4.
• SAR adds 0.03 GFLOPs and 1.56M parameters (2.3%) to HRNet at 1024x1024 input with no test-time cost.

Measured effects reported in the paper

Surprising findings

• Random pre-defined anchors (standard normal, orthogonal, max-equiangular) give unstable gains, and CAR collapses to 0.0 IoU on the rarest 'Mouse' class while SAR reaches 40.3.
• On CIFAR-100-LT, CAR reduces accuracy relative to the MixUp baseline (-1.5 stage 1), while SAR improves it.

Limitations

• Not applied to object detection or query-based segmentation methods.
• Semantic anchors not used at test time, so potential test-time benefits are unexplored.
• Gains on several strong baselines are modest (e.g., +0.4 on SegFormer and DeepLabV3 Pascal-Context).

Experimental setup

Datasets: Cityscapes ADE20K Pascal-Context CIFAR-100 CUB-200-2011 CIFAR-100-LT ImageNet-1K

Benchmarks: Cityscapes ADE20K Pascal-Context CIFAR-100 CUB-200 CIFAR-100-LT

Metrics: mIoU Top-1 accuracy head/body/tail mIoU per-class IoU

Baselines: HRNet FCN DeepLabV3 OCRNet SegFormer UPerNet ProtoSeg CAR DisAlign MiSLAS

Builds on: prototype-based learning CAR MiSLAS DisAlign mmsegmentation

Closest papers by meaning

• FedSA: A Unified Representation Learning via Semantic Anchors for Prototype-based Federated Learning (AAAI 2025 · similarity 0.87)
• Beyond Federated Prototype Learning: Learnable Semantic Anchors with Hyperspherical Contrast for Domain-Skewed Data (AAAI 2025 · similarity 0.86)
• Generative Modeling of Class Probability for Multi-Modal Representation Learning (CVPR 2025 · similarity 0.85)
• Hunting Attributes: Context Prototype-Aware Learning for Weakly Supervised Semantic Segmentation (CVPR 2024 · similarity 0.85)
• Self-Supervised Representation Learning with Meta Comprehensive Regularization (AAAI 2024 · similarity 0.84)
• Hierarchical Semantic Tree Anchoring for CLIP-Based Class-Incremental Learning (NeurIPS 2026 · similarity 0.84)