Research map / Detection and segmentation / CNN features
UNIP: Rethinking Pre-trained Attention Patterns for Infrared Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Benchmarks six RGB pre-training methods for ViT on infrared semantic segmentation, analyzes attention patterns via NMI, and proposes UNIP (NMI-guided hybrid attention distillation, InfMix dataset, LL-FPN) to boost small ViTs.
Problem
Transfer of RGB-pre-trained vision transformers to infrared semantic segmentation is underexplored, and it is unclear which pre-training methods and attention patterns help under a large domain gap with limited labeled data.
Approach
Identify hybrid local-global attention patterns (quantified by query-key NMI) as crucial for segmentation, then distill the hybrid-pattern attention map from a large teacher into a small ViT on a mixed RGB/infrared dataset, with a last-layer FPN for fine-tuning.
Key results
- UNIP-S (MAE-L teacher) reaches 64.37% average fine-tuning mIoU, matching MAE-L (64.35%) at about 1/10 computational cost, and gaining 8.98% over MAE-S.
- UNIP-S (MAE-L teacher) scores 71.35 on SODA and 53.76 on MFNet-T, surpassing Mask2Former (67.58 / 51.30) and TINN (69.45 / 51.93) on the same comparison table.
- With iBOT-L teacher, UNIP-B reaches 51.58% average linear probing mIoU, exceeding its teacher iBOT-L (49.51%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| distill hybrid attention with NMI-HAD, using InfMix and LL-FPN rather than the MAE-S baseline | MAE-S-sized ViT · Infrared average (SODA, MFNet-T, SCUT-Seg) | Average FT mIoU: 55.39 -> 64.37 (+8.98) | Helped |
| distill hybrid attention with NMI-HAD, using InfMix and LL-FPN rather than the MAE-S baseline | MAE-S-sized ViT · Infrared average (SODA, MFNet-T, SCUT-Seg) | Average LP mIoU: 31.15 -> 44.04 (+12.89) | Helped |
| replace feature distillation with attention distillation at Layer 18 (ablation) | UNIP-S distilled from MAE-L · Infrared average | Average FT mIoU: 60.22 -> 64.37 | Helped |
| use Layer 18 rather than Layer 24 as the MAE-L attention target (ablation) | UNIP-S distilled from MAE-L · Infrared average | Average FT mIoU: 62.38 -> 64.37 | Helped |
| add LL-FPN for the hybrid-pattern MAE-L target (ablation) | UNIP-S distilled from MAE-L Layer 18 · Infrared average | Average FT mIoU: 63.45 -> 64.37 | Helped |
| add LL-FPN for the local-pattern MAE-L target (ablation) | UNIP-S distilled from MAE-L Layer 24 · Infrared average | Average FT mIoU: 62.08 -> 62.38 | Helped |
| add LL-FPN for the hybrid-pattern DINO-B target (ablation) | UNIP-S distilled from DINO-B Layer 9 · Infrared average | Average FT mIoU: 62.16 -> 63.00 | Helped |
| add LL-FPN for the global-pattern DINO-B target (ablation) | UNIP-S distilled from DINO-B Layer 12 · Infrared average | Average FT mIoU: 61.90 -> 62.19 | Helped |
| add the ImageNet-1K subset to InfMix (ablation) | UNIP-S distilled from MAE-L · Infrared average | Average FT mIoU: 63.58 -> 64.37 | Helped |
| add COCO images to InfMix (ablation) | UNIP-S distilled from MAE-L · Infrared average | Average FT mIoU: 63.50 -> 64.37 | Helped |
| convert ImageNet and COCO images to grayscale in InfMix (ablation) | UNIP-S distilled from MAE-L · Infrared average | Average FT mIoU: 63.84 -> 64.37 | Helped |
| use mixed InfMix rather than ImageNet-1K alone (ablation) | UNIP-S distilled from MAE-L · Infrared average | Average FT mIoU: 62.71 -> 64.37 | Helped |
Surprising findings
- ImageNet accuracy and fine-tuning accuracy do not reliably predict infrared segmentation transfer; ImageNet FT vs infrared FT correlation is 0.08 on average.
- Small MIM models (MAE-S/B) lag supervised and contrastive models on infrared segmentation, while MAE-L is competitive, indicating the hybrid attention pattern emerges only in larger MIM models.
Limitations
- Validated only for semantic segmentation in the infrared domain due to limited compute; other dense tasks and modalities are untested.
- Benchmark is restricted to ViT models pre-trained on ImageNet with vanilla architecture, excluding larger-data models such as DINOv2 and EVA.
- The empirical NMI threshold s=0.09 and the target layers are chosen heuristically per teacher.
Experimental setup
Datasets: InfPre ImageNet-1K COCO SODA MFNet-T SCUT-Seg ADE20K MFNet-RGB NYUDepthv2 SUN-RGBD
Benchmarks: SODA MFNet-T SCUT-Seg ADE20K NYUDepthv2 SUN-RGBD ImageNet-1K
Metrics: mIoU fine-tuning mIoU linear probing mIoU Pearson correlation NMI
Baselines: DeiT DeiT III DINO iBOT MAE CrossMAE Mask2Former TINN MCNet PAD SegFormer UPerNet
Builds on: masked autoencoders ViTDet UperNet DINO iBOT
Closest papers by meaning
- Integrating Low-Level Visual Cues for Enhanced Unsupervised Semantic Segmentation (AAAI 2025 · similarity 0.87)
- IF-Bench: Benchmarking and Enhancing MLLMs for Infrared Images with Generative Visual Prompting (CVPR 2026 · similarity 0.87)
- Unsupervised Semantic Segmentation Through Depth-Guided Feature Correlation and Sampling (CVPR 2024 · similarity 0.87)
- UniDSeg: Unified Cross-Domain 3D Semantic Segmentation via Visual Foundation Models Prior (NeurIPS 2024 · similarity 0.87)
- Label-Efficient Few-Shot Semantic Segmentation with Unsupervised Meta-Training (AAAI 2024 · similarity 0.87)
- Spectral Prompt Tuning: Unveiling Unseen Classes for Zero-Shot Semantic Segmentation (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.