Research map / Detection and segmentation / CNN features
SCTNet: Single-Branch CNN with Transformer Semantic Information for Real-Time Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SCTNet, a single-branch CNN for real-time segmentation that uses a training-only SegFormer semantic branch, aligned via CFBlock and SIAM (CWD loss), so only the CNN is deployed at inference.
Problem
Real-time semantic segmentation methods rely on an extra semantic branch for long-range context, which adds inference cost and slows speed; single-branch lightweight CNNs lack long-range semantics and lose accuracy.
Approach
Train a single-branch CNN to align its features with a training-only transformer (SegFormer) semantic branch via a Conv-Former block that mimics transformer structure with convolutions only, plus a semantic information alignment module (backbone feature alignment and shared decoder head alignment) using channel-wise distillation loss; the transformer is discarded at inference.
Key results
- SCTNet-B-Seg100 achieves 80.5% mIoU at 62.8 FPS (Torch) on Cityscapes val, measured on RTX 3090.
- SCTNet-B achieves 43.0% mIoU at 145.1 FPS on ADE20K, about 1.6x faster than RTFormer-B with 0.9% higher mIoU.
- SCTNet-B reaches 35.9% mIoU at 141.5 FPS on COCO-Stuff-10K, 0.6% above RTFormer-B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add CFBlock in place of ResBlock (ablation) | SCTNet · Cityscapes Seg100 | mIoU(%): 76.4 -> 78.5 (+2.1) | Helped |
| add backbone feature alignment (BFA) (ablation) | SCTNet with CFBlock · Cityscapes Seg100 | mIoU(%): 78.5 -> 79.7 (+1.2) | Helped |
| add shared decoder head alignment (SDHA) (ablation) | SCTNet with BFA · Cityscapes Seg100 | mIoU(%): 79.7 -> 80.5 (+0.8) | Helped |
| add SIAM (ablation) | SCTNet-B · Cityscapes Seg100 | mIoU(%): 78.5 -> 80.5(+2.0) | Helped |
| replace ResBlock with CFBlock (ablation) | real-time segmentation network · Cityscapes, block comparison | mIoU(%): 77.9 -> 79.4 | Helped |
| use 1 × 7 and 7 × 1 stripe kernels rather than a single 1 × 1 convolution (ablation) | CFBlock · Cityscapes, no pre-training | mIoU(%): 78.4 -> 79.4 | Helped |
| use L2 alignment loss instead of the baseline loss (ablation) | SCTNet · Cityscapes Seg100 | mIoU: 78.5 -> 77.8 | Hurt |
| use KL alignment loss instead of the baseline loss (ablation) | SCTNet · Cityscapes Seg100 | mIoU: 78.5 -> 78.6 | No clear effect |
| use CWD alignment loss (ablation) | SCTNet · Cityscapes Seg100 | mIoU: 78.5 -> 79.7 | Helped |
| apply alignment loss at logits, decoder, stage4, and stage3 (ablation) | SCTNet · Cityscapes | mIoU(%): 78.2 -> 79.7(+1.5) | Helped |
| increase alignment loss weights from 3,15,15,15 to 3,20,20,20 (ablation) | SCTNet · Cityscapes Seg100 | mIoU: 79.7 -> 79.1 | Hurt |
| vary the training-only semantic branch among OCRNet, InternImage, SwinTransformer, and SegFormer (ablation) | SCTNet · Cityscapes Seg100 | mIoU: 80.0; 80.5; 79.8; 80.5 | No clear effect |
Surprising findings
- Adding SIAM to DDRNet-23 and SeaFormer gives only slim gains (+0.1 and +0.4 mIoU), which the authors attribute to their existing semantic branches.
Limitations
- Training requires a pretrained SegFormer teacher and extra training cost, which the paper does not quantify in detail.
- Gains from SIAM are smaller on architectures that already contain a semantic branch (SeaFormer, DDRNet).
Experimental setup
Datasets: ImageNet-1K Cityscapes ADE20K COCO-Stuff-10K
Benchmarks: Cityscapes ADE20K COCO-Stuff-10K
Metrics: mIoU FPS parameters
Baselines: BiSeNetV2 STDC DDRNet RTFormer SeaFormer TopFormer SegNext AFFormer PIDNet SFNet
Builds on: SegFormer GFA (GPU-Friendly Attention) CWD (channel-wise distillation) DDRNet DAPPM
Closest papers by meaning
- Rethinking Semantic Segmentation From a Sequence-to-Sequence Perspective With Transformers (CVPR 2021 · similarity 0.88)
- RTFormer: Efficient Design for Real-Time Semantic Segmentation with Transformer (NeurIPS 2022 · similarity 0.88)
- PIDNet: A Real-Time Semantic Segmentation Network Inspired by PID Controllers (CVPR 2023 · similarity 0.87)
- Graph-Guided Architecture Search for Real-Time Semantic Segmentation (CVPR 2020 · similarity 0.87)
- SasWOT: Real-Time Semantic Segmentation Architecture Search WithOut Training (AAAI 2024 · similarity 0.86)
- Golden Cudgel Network for Real-Time Semantic Segmentation (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.