Research map / Detection and segmentation / CNN features
Semantic-Aware Transformation-Invariant RoI Align
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Semantic RoI Align (SRA), a two-stage detector RoI feature extractor that learns semantic sampling masks from RoI features, with a Dynamic Feature Sampler and Area Embedding, improving COCO AP over RoI Align.
Problem
RoI feature extractors in two-stage detectors achieve invariance only under limited transformations (scale, rigid rotation), and existing deformable variants fail to generalize to rotation and pose transformations.
Approach
Learn N semantic sampling masks per RoI via a semantic attention structure (RoI descriptor plus local semantic features plus position embedding) and pool features as weighted sums under these masks, so sampling follows semantic parts under transformations; add a Dynamic Feature Sampler that picks aspect-ratio-consistent sub-sampling sizes and an Area Embedding encoding the sampled area.
Key results
- SRA with Faster R-CNN ResNet-50 reaches 39.2 AP on COCO val vs 37.5 for RoI Align, with only 0.2M extra parameters and about 1.1% more FLOPs.
- On COCO test-dev with Faster R-CNN R-50, SRA gets 39.2 AP vs 37.7 for RoI Align and 38.1 for DRoIPooling, and exceeds RoIAttn (38.0) by 1.2 AP.
- SRA improves RoI Align across detectors and backbones on COCO test-dev, e.g. +1.0 AP with Mask R-CNN Swin-T and +1.6 AP with PVTv2-B0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace RoI Align with SRA | Faster R-CNN, ResNet-50 · MS COCO val2017 | AP: 37.5 -> 39.2 | Helped |
| Replace RoI Align with SRA | Faster R-CNN, ResNet-50 · MS COCO test-dev | AP: 37.7 -> 39.2 | Helped |
| Replace RoI Align with SRA | FGFA + Faster R-CNN, ResNet-50 · ImageNet VID val | AP: 47.1 -> 48.0 | Helped |
| Replace RoI Align with SRA | Faster R-CNN, ResNet-50 · Pascal VOC 2007 test | AP: 80.2 -> 81.8 | Helped |
| Add a semantic feature map to SRA mask regression (ablation) | SRA · MS COCO val2017 | AP: 36.4 -> 38.2 | Helped |
| Add Area Embedding (ablation) | SRA · MS COCO val2017 | AP: 36.4 -> 39.2 | Helped |
| Replace position embedding with Area Embedding (ablation) | SRA · MS COCO val2017 | AP: 38.8 -> 39.2 | Helped |
| Use average rather than concatenation for the RoI descriptor regressor (ablation) | SRA · MS COCO val2017 | AP: 38.9 -> 39.2 | Helped |
| Use average rather than maximum for the RoI descriptor regressor (ablation) | SRA · MS COCO val2017 | AP: 39.0 -> 39.2 | Mixed |
| Set amplification factor gamma to 50 rather than 1, 5, or 500 (ablation) | SRA · MS COCO val2017 | AP: 38.3, 38.7, 39.2, 36.8 | Mixed |
| Increase the number of semantic masks from 9 or 25 to 49 (ablation) | SRA · MS COCO val2017 | AP: 38.0 -> 39.2; 38.6 -> 39.2 | Helped |
| Use 100 rather than 49 semantic masks (ablation) | SRA · MS COCO val2017 | AP: 39.2 -> 39.4 | Helped |
Surprising findings
- Modulated DRoIPooling gains 1.3 AP over SRA's smaller model size is not matched; DRoIPooling has far more parameters (149.4M) than SRA (42.0M) yet lower AP.
- Replacing the learned masks with a convolution-based mask predictor (w/ Conv.) drops AP below the RoI Align baseline (36.4 vs 37.5).
Limitations
- Computational overhead increases with the number of masks and feature sampling resolution.
- Invariance gains are measured mainly on COCO; generalization to other transformation types is evaluated only through cosine similarity tests.
Experimental setup
Datasets: MS COCO Pascal VOC ImageNet VID
Benchmarks: MS COCO test-dev MS COCO val2017 Pascal VOC 2007 test ImageNet VID validation
Metrics: AP AP50 AP75 APS APM APL FLOPs parameters cosine similarity of RoI features
Baselines: RoI Pooling RoI Align DRoIPooling Adaptive RoI Align Precise RoI Align RoIAttn RoI Transformer
Builds on: RoI Align Deformable RoI Pooling Faster R-CNN RoIAttn RoI Transformer
Closest papers by meaning
- Semantic-Aligned Fusion Transformer for One-Shot Object Detection (CVPR 2022 · similarity 0.88)
- Accelerating DETR Convergence via Semantic-Aligned Matching (CVPR 2022 · similarity 0.88)
- Breaking Alignment Barriers: TPS-Driven Semantic Correlation Learning for Alignment-Free RGB-T Salient Object Detection (AAAI 2026 · similarity 0.87)
- Feature Selective Networks for Object Detection (CVPR 2018 · similarity 0.87)
- Temporal ROI Align for Video Object Recognition (AAAI 2021 · similarity 0.86)
- Improving Single Domain-Generalized Object Detection: A Focus on Diversification and Alignment (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.