Research map / Detection and segmentation / CNN features
Parameter-Inverted Image Pyramid Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PIIP, a multi-branch image pyramid where higher-resolution inputs go to smaller pretrained ViTs and lower-resolution inputs to larger ones, linked by cross-branch deformable-attention interactions, cutting compute while matching or beating single-branch ViTs on detection, segmentation and classification.
Problem
Image pyramids run the same large model on every resolution, so compute grows quickly with input resolution, making multi-scale inference for pretrained vision foundation models expensive.
Approach
Pair pretrained ViT branches of decreasing size with increasing image resolution (parameter-inverted), and connect adjacent branches with zero-initialized deformable cross-attention interaction units so that the branches complement each other while reusing pretrained weights.
Key results
- PIIP-TSB reaches 46.6 box AP and 41.4 mask AP at 453G FLOPs with Mask R-CNN 1x, matching ViTDet-B (43.8 AP) at similar cost while using fewer FLOPs than ViTDet-B's 463G baseline for lower-resolution variants.
- On InternViT-6B, PIIP-LH6B reaches 55.7 box AP (vs 53.8) using 13911G vs 24418G FLOPs, and 59.65 mIoU on ADE20K (vs 58.36) at about 58% of the FLOPs.
- PIIP-SBL achieves 54.5 box AP with Cascade R-CNN 3x+MS and 57.9 box AP with DINO, reported as competitive with prior large backbones.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use parameter-inverted rather than parameter-direct branch sizes (ablation) | PIIP-TSB · COCO val2017, Mask R-CNN 1× | AP^b: 42.6 -> 46.6 | Helped |
| Use parameter-inverted rather than parameter-direct branch sizes (ablation) | PIIP-TSB · COCO val2017, Mask R-CNN 1× | AP^m: 36.5 -> 41.4 | Helped |
| Add an image pyramid to single-branch ViT-B multi-scale training (ablation) | ViT-B · COCO val2017, Mask R-CNN 1× | AP^b: 43.8 -> 44.8 | Helped |
| Add interactions to the BBB image pyramid (ablation) | BBB · COCO val2017, Mask R-CNN 1× | AP^b: 43.3 -> 44.5 | Helped |
| Add interactions to the BBB image pyramid (ablation) | BBB · COCO val2017, Mask R-CNN 1× | AP^m: 37.9 -> 38.7 | Helped |
| Increase deformable cross-attention interactions from 0 to 12 (ablation) | PIIP-TSB · COCO val2017, Mask R-CNN 1× | AP^b: 41.3 -> 43.9 | Helped |
| Use deformable instead of regular attention with 12 interactions (ablation) | PIIP-TSB · COCO val2017, Mask R-CNN 1× | AP^b: 42.0 -> 43.9 | Helped |
| Merge outputs from all branches rather than only the B branch (ablation) | PIIP-TSB · COCO val2017, Mask R-CNN 1× | AP^b: 43.1 -> 46.6 | Helped |
| Initialize ViT-B/L branches with BEiTv2 rather than DeiT III (ablation) | PIIP-SBL · COCO val2017 | AP^b: 50.0 -> 51.8 | Helped |
| Use PIIP-TSB rather than ViT-B | ADE20K UperNet · ADE20K | mIoU: 51.0 -> 51.6 | Helped |
| Use PIIP-SBL rather than ViT-L | ADE20K UperNet · ADE20K | mIoU: 53.6 -> 54.3 | Helped |
| Use PIIP-LH6B rather than single-branch InternViT-6B | InternViT-6B · COCO val2017, Mask R-CNN 1× | AP^b: 53.8 -> 55.7 | Helped |
Surprising findings
- Simply raising ViTDet-L's input resolution to 1792 gives 48.3 AP at about 4x the FLOPs, which PIIP-TSBL beats (49.6 AP) at 1535G, suggesting gains are not only from resolution.
- A parameter-direct configuration (TSB with 448/672/896 resolutions) performs worse (42.6 AP) than the parameter-inverted one (46.6 AP).
Limitations
- Memory consumption is higher than single-branch models due to increased parameter count.
- Only ViT-based models are studied; hierarchical CNN or heterogeneous branches are unexplored.
Experimental setup
Datasets: ImageNet-1K ImageNet-21K COCO ADE20K
Benchmarks: COCO val2017 ADE20K ImageNet-1K
Metrics: box AP mask AP mIoU top-1 accuracy FLOPs
Baselines: ViTDet-B ViTDet-L ViT-Adapter InternViT-6B Swin ConvNeXt InternImage DeiT-B ViT-L
Builds on: ViTDet ViT-Adapter Deformable attention CrossViT Feature Pyramid Network
Closest papers by meaning
- Feature Pyramid Networks for Object Detection (CVPR 2017 · similarity 0.87)
- Scale-Equalizing Pyramid Convolution for Object Detection (CVPR 2020 · similarity 0.85)
- M2Det: A Single-Shot Object Detector Based on Multi-Level Feature Pyramid Network (AAAI 2019 · similarity 0.85)
- Pyramid Architecture for Multi-Scale Processing in Point Cloud Segmentation (CVPR 2022 · similarity 0.84)
- Integrally Pre-Trained Transformer Pyramid Networks (CVPR 2023 · similarity 0.83)
- An Analysis of Scale Invariance in Object Detection SNIP (CVPR 2018 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.