Research map / Detection and segmentation / CNN features
FeatUp: A Model-Agnostic Framework for Features at Any Resolution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FeatUp, a model-agnostic framework that upsamples deep features using multi-view consistency over jittered low-resolution feature views, via either a learned JBU-based feedforward upsampler or a per-image implicit network.
Problem
Deep features from vision backbones are spatially coarse (e.g., ResNet-50 yields 7x7 maps; ViTs have 14x14 for 224 input), which limits their use for dense prediction and explanation methods without losing semantics.
Approach
Learn a high-resolution feature map whose learned downsampled, identically-jittered versions reproduce the backbone's low-resolution features across small image transforms (NeRF-like multi-view consistency), with a learned uncertainty-weighted likelihood loss; the upsampler is a learnable generalization of Joint Bilateral Upsampling (with a CUDA kernel) or a per-image implicit MLP with Fourier fea
Key results
- FeatUp (Implicit) achieves the best CAM scores (A.D. 8.84, A.I. 5.60) and linear-probe COCO-Stuff segmentation (71.58% accuracy, 47.37 mIoU) versus 65.17% / 40.65 for low-res ViT features.
- With SegFormer on ADE20K, FeatUp (JBU) reaches 44.2 mIoU, above FADE (43.6), CARAFE (42.4) and bilinear (39.7), with +0.16M parameters.
- The CUDA JBU kernel uses up to two orders of magnitude less memory and is up to 10x faster than a PyTorch Unfold implementation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FeatUp (JBU) instead of the strongest listed baseline | ViT-S/16 features · CAM Score | A.D.: 10.24 -> 9.83 | Helped |
| Use FeatUp (Implicit) instead of the strongest listed baseline | ViT-S/16 features · CAM Score | A.I.: 5.16 -> 5.60 | Helped |
| Use FeatUp (Implicit) instead of the strongest listed baseline | ViT-S/16 features · COCO-Stuff val | accuracy: 67.72 -> 71.58 | Helped |
| Use FeatUp (Implicit) instead of the strongest listed baseline | ViT-S/16 features · COCO-Stuff val | mIoU: 42.39 -> 47.37 | Helped |
| Use FeatUp (Implicit) instead of the strongest listed baseline | ViT-S/16 features · Depth Estimation | RMSE: 1.09 -> 1.04 | Helped |
| Use FeatUp (JBU) instead of the strongest listed baseline | ViT-S/16 features · Depth Estimation | δ > 1.25: 0.920 -> 0.938 | Helped |
| Use FeatUp (JBU) instead of FADE | SegFormer · ADE20K val | mIoU: 43.6 -> 44.2 | Helped |
| Use FeatUp (JBU) instead of SAPA | SegFormer · ADE20K val | mAcc: 55.3 -> 55.8 | Helped |
| Remove the guidance-signal MLP (ablation) | FeatUp (JBU) with SegFormer · ADE20K val | mIoU: 44.2 -> 42.9 | Hurt |
| Replace softmax with Euclidean distance (ablation) | FeatUp (JBU) with SegFormer · ADE20K val | mIoU: 44.2 -> 43.8 | Hurt |
| Replace softmax with cosine distance (ablation) | FeatUp (JBU) with SegFormer · ADE20K val | mIoU: 44.2 -> 43.7 | Hurt |
| Remove the attention downsampler (ablation) | FeatUp (Implicit) · COCO-Stuff val | mIoU: 47.37 -> 46.79 | Hurt |
Surprising findings
- Simple bilinear and resize-conv upsampling of ViT features are competitive with or better than several learned upsamplers on linear-probe depth RMSE (resize-conv 1.14 vs bilinear 1.19), while strided and large-image baselines hurt depth RMSE substantially.
Limitations
- Implicit variant can produce halo effects around objects.
- Relies on the input image's spatial signal, so unrelated patterns (e.g., rug texture) can transfer into feature maps.
- Per-image implicit variant requires inference-time optimization for each image.
Experimental setup
Datasets: ImageNet COCO-Stuff MiDaS pseudo-labels ADE20K
Benchmarks: COCO-Stuff ImageNet ADE20K MiDaS depth pseudo-labels
Metrics: Average Drop Average Increase mIoU pixel accuracy RMSE delta > 1.25 mAcc aAcc
Baselines: Bilinear upsampling Resize-conv Strided convolution Large image CARAFE SAPA FADE IndexNet A2U DIP
Builds on: Joint Bilateral Upsampling NeRF LIIF CARAFE SegFormer
Closest papers by meaning
- A Simple yet Universal Framework for Depth Completion (NeurIPS 2024 · similarity 0.84)
- FeatSharp: Your Vision Model Features, Sharper (ICML 2025 · similarity 0.84)
- Know Your Neighbors: Improving Single-View Reconstruction via Spatial Vision-Language Reasoning (CVPR 2024 · similarity 0.84)
- Win-Win: Training High-Resolution Vision Transformers from Two Windows (ICLR 2024 · similarity 0.84)
- Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data (CVPR 2024 · similarity 0.84)
- Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.