Research map / Vision-language and multimodal / Human and embodied modeling
FeatSharp: Your Vision Model Features, Sharper
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FeatSharp, a lightweight module that upsamples low-resolution vision encoder feature maps by fusing JBU-upsampled features with tiled high-res features via a local-attention block, and uses it to build hi-res teachers for training FeatSharp-RADIO.
Problem
Vision encoders such as CLIP, SigLIP and DFN CLIP produce low-resolution feature maps and do not scale well to higher input resolutions, limiting fine-grained detail for dense perception and for distillation targets in agglomerative models.
Approach
Extend FeatUp's JBU multi-view-consistency upsampler with a learned bias buffer and a single local-attention+SwiGLU refinement block that fuses the JBU-upsampled map with a mosaic of tile features from the same frozen encoder, so new detail visible only at higher resolution is recovered at modest cost.
Key results
- FeatSharp 2x upsampling with 2x input size reaches 53.13 mIoU on ADE20k linear probe with RADIOv2.5-L, +1.66 over the published 51.47.
- On COCO object detection with RADIOv2.5-L, FeatSharp 2x achieves 54.83 AP vs 51.38 baseline and 46.71 for FeatUp.
- FeatSharp-RADIO improves MTL Delta_m by +0.39% over the reproduction baseline, and is the only upsampler categorically better across tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the RADIOv2.5-L baseline with 2× FeatSharp upsampling and 2× input size | RADIOv2.5-L semantic segmentation · ADE20k | mIoU: 51.47 -> 53.13; +1.66 mIoU | Helped |
| use FeatSharp instead of other upsampling methods | object detection with RADIOv2.5-L · COCO 2017 | AP: 51.38 -> 54.83 | Helped |
| use FeatSharp instead of other upsampling methods | object detection with SigLIP2-SO400M-512 · COCO 2017 | AP Sm: 30.31 -> 36.85 | Helped |
| use FeatSharp-upsampled teacher features during RADIO training | RADIO · RADIO benchmark suite | Δm %: +0.39% | Helped |
| use tiled teacher features only | RADIO training · RADIO benchmark suite | Δm %: -0.27 | Hurt |
| use FeatUp-upsampled teacher features | RADIO training · RADIO benchmark suite | Δm %: -0.13 | Hurt |
| add the learned de-bias buffer before upsampling (ablation) | FeatSharp and FeatUp · multi-view consistency across tested featurizers | fidelity: FeatSharp 2x: 0.020, 0.121, 0.017, 0.208, 0.067, 0.014, 0.014 | Helped |
| use JBU rather than bilinear for the residual pathway, with tiles (ablation) | FeatSharp 4× · SigLIP | Fidelity: 1.522 -> 1.580 | Helped |
| use JBU rather than bilinear for the residual pathway, with tiles (ablation) | FeatSharp 4× · RADIO | Fidelity: 5.921 -> 5.898 | No clear effect |
| use attention alongside the MLP rather than MLP alone (ablation) | FeatSharp with the Long recipe · 2× upsampling on RADIO | Fidelity: 5.397 -> 5.502 | Helped |
| use attention alongside the MLP rather than MLP alone (ablation) | FeatSharp with the Long recipe · 2× upsampling on SigLIP | Fidelity: 1.581 -> 1.584 | Helped |
| increase local attention window size (ablation) | FeatSharp · window-size ablation | Fidelity | No clear effect |
Surprising findings
- FeatUp's JBU upsampler degraded object detection AP for RADIOv2.5-L and SigLIP2 relative to bilinear upsampling.
- The best configuration for DFN CLIP Probe3D depth was a FeatUp JBU stack with random weights, suggesting structural priors drive gains.
Limitations
- Tiling requires multiple encoder evaluations, adding compute relative to single-pass upsampling.
- 3x upsampling was generally slightly worse than 2x or 4x and was not explained.
- Whether FeatSharp-RADIO avoids mode switching at high resolution remains unclear according to the authors.
Experimental setup
Datasets: SA-1B COCO ADE20k COCO 2017
Benchmarks: ADE20k COCO 2017 ImageNet-1k Probe3D Pascal Context NYUDv2 VILA MME MMMU SEED
Metrics: multi-view fidelity mIoU AP MTL Delta_m accuracy RMSE
Baselines: FeatUp Bilinear upsampling SAPA ReSFU Tiling S2 RADIOv2.5-L
Builds on: FeatUp AM-RADIO PHI-S ViTDet ViT-Denoiser
Closest papers by meaning
- Perception Encoder: The best visual embeddings are not at the output of the network (NeurIPS 2025 · similarity 0.84)
- FeatUp: A Model-Agnostic Framework for Features at Any Resolution (ICLR 2024 · similarity 0.84)
- Vision Transformers Need Registers (ICLR 2024 · similarity 0.84)
- UMFC: Unsupervised Multi-Domain Feature Calibration for Vision-Language Models (NeurIPS 2024 · similarity 0.84)
- VITRIX-UniViTAR: Unified Vision Transformer with Native Resolution (NeurIPS 2025 · similarity 0.84)
- On the Surprising Effectiveness of Attention Transfer for Vision Transformers (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.