Research map / Vision-language and multimodal / Human and embodied modeling
JAFAR: Jack up Any Feature at Any Resolution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces JAFAR, a lightweight cross-attention feature upsampler that uses the image as high-resolution guidance and SFT-modulated keys, trained task-agnostically at low ratios to generalize to arbitrary output resolutions.
Problem
Foundation vision encoders produce low-resolution features (14-16x downsampling), and existing feature upsamplers are either task-specific, fixed-scale, over-smoothed, or require per-image optimization.
Approach
An attention-based upsampler whose queries come from low-level high-res image features and whose keys are semantically enriched via SFT modulation from low-res encoder features; trained with a multi-resolution cosine+L2 alignment loss at low upsampling ratios, generalizing to much higher scales at inference.
Key results
- On linear-probe segmentation with DINOv2 ViT-S/14, JAFAR reaches 84.44 mIoU on VOC and 61.47 on Cityscapes, the best among all baselines.
- On Grad-CAM evaluation, JAFAR achieves ADCC 73.3 versus 64.3 for the next-best method (FeatUp), a gain of about 8 points.
- JAFAR improves BeV vehicle IoU by up to about 5 points over no upsampling (e.g., PointBeV 34.89 to 37.20).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use JAFAR instead of the strongest reported baseline | DINOv2 ViT-S/14 linear-probe segmentation · Cityscapes | mIoU: 56.06 -> 61.47 | Helped |
| Use JAFAR instead of ReSFU | DINOv2 ViT-S/14 depth estimation · COCO | δ1: 66.14 -> 62.18 | Hurt |
| Use JAFAR instead of the second-best Strided method | Grad-CAM · ImageNet validation | ADCC: 65.3 -> 73.3 | Helped |
| Use JAFAR instead of FeatUp | CLIP ViT-B/16 zero-shot segmentation · VOC | mIoU: 32.27 -> 35.70 | Helped |
| Use JAFAR instead of Bilinear | nuScenes SimpleBeV · vehicle segmentation | mIoU: 33.67 -> 36.59 | Helped |
| Add self-distillation (ablation) | JAFAR · VOC semantic segmentation (448) | mIoU: 61.49 -> 62.30 | Helped |
| Remove SFT modulation from keys (ablation) | JAFAR · Cityscapes semantic segmentation | mIoU: 61.47 -> 56.53 | Hurt |
| Replace SFT-modulated keys with linearly projected encoder features (ablation) | JAFAR · Cityscapes semantic segmentation | mIoU: 61.47 -> 52.45 | Hurt |
| Replace SFT modulation with concatenated semantic features and preliminary keys (ablation) | JAFAR · Cityscapes semantic segmentation | mIoU: 61.47 -> 58.70 | Hurt |
| Add RoPE positional encoding (ablation) | JAFAR · Cityscapes semantic segmentation (448) | mIoU: 36.09 -> 60.86 | Helped |
| Set attention-head count to 1 instead of 4 (ablation) | JAFAR · VOC semantic segmentation | mIoU: 84.44 -> 84.13 | Hurt |
| Set attention-head count to 2 instead of 4 (ablation) | JAFAR · VOC semantic segmentation | mIoU: 84.44 -> 84.27 | Hurt |
Surprising findings
- Training only at low ratios and low resolutions (8x8 to 32x32, up to 4x) generalizes to 896x896 outputs without high-resolution supervision.
- Iterative LiFT (stacking two x2 upsamplers) does not outperform a single LiFT x2 upsampler followed by bilinear interpolation.
Limitations
- Global attention cost and memory grow with key count, becoming prohibitive for large key sets (e.g., 64x64 or higher).
- Requires training a separate upsampler for each backbone.
Experimental setup
Datasets: ImageNet COCO-Stuff ADE20K Pascal VOC Cityscapes nuScenes ImageNet validation
Benchmarks: COCO-Stuff Pascal VOC ADE20K Cityscapes NYU/COCO depth estimation ImageNet CAM evaluation nuScenes BeV segmentation
Metrics: mIoU pixel accuracy delta_1 RMSE A.D A.I A.G Coherency Complexity ADCC
Baselines: Bilinear Nearest Large Image Strided CARAFE SAPA DySample ReSFU FeatUp LiFT LoftUp
Builds on: LiFT FeatUp SFT RoPE MaskCLIP
Closest papers by meaning
- Win-Win: Training High-Resolution Vision Transformers from Two Windows (ICLR 2024 · similarity 0.84)
- FeatUp: A Model-Agnostic Framework for Features at Any Resolution (ICLR 2024 · similarity 0.83)
- HIIF: Hierarchical Encoding based Implicit Image Function for Continuous Super-resolution (CVPR 2025 · similarity 0.83)
- FIPER: Factorized Features for Robust Image Super-Resolution and Compression (NeurIPS 2025 · similarity 0.83)
- GaussianSR: High Fidelity 2D Gaussian Splatting for Arbitrary-Scale Image Super-Resolution (AAAI 2025 · similarity 0.82)
- BA-SAM: Scalable Bias-Mode Attention Mask for Segment Anything Model (CVPR 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.