Research map / Architectures and efficiency / Attention and transformers
MSPE: Multi-Scale Patch Embedding Prompts Vision Transformers to Any Resolution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MSPE, which replaces a ViT's patch embedding with multiple adaptive, PI-resized convolution kernels trained for 5 epochs with a frozen encoder, so pretrained ViTs can process images at native variable resolutions.
Problem
Pretrained ViTs are tied to a fixed input resolution; changing resolution degrades accuracy and existing resolution-adaptive methods either require costly full retraining or are incompatible with pretrained models.
Approach
Keep the Transformer encoder frozen and optimize only the patch embedding layer: multiple variable-sized kernels whose size adapts to input resolution via pseudo-inverse resize, trained with mixed-resolution loss so images are processed at native size and aspect ratio without resizing.
Key results
- On ImageNet-1K at 28x28 resolution with ViT-B, MSPE reaches 56.41% top-1 vs 14.86% for FlexiViT and 8.52% for vanilla ViT.
- On ADE20K at 128x128 (SETR), MSPE achieves 39.65 mIOU vs 31.44 for FlexiViT and 2.89 for vanilla.
- MSPE trains for only 5 epochs (about 3 hours on ImageNet-1K) versus ResFormer's 200 epochs (over 50 hours).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace standard patch embedding with multi-scale adaptive kernels | ViT-B · ImageNet-1K, 28×28 | Top-1 accuracy: 14.86 -> 56.41 | Helped |
| Replace standard patch embedding with multi-scale adaptive kernels | SETR Naive (ViT-L) · ADE20K, 128×128 | mIOU: 31.44 -> 39.65 | Helped |
| Replace standard patch embedding with multi-scale adaptive kernels | SETR Naive (ViT-L) · Cityscapes, 192×192 | mIOU: 59.78 -> 65.90 | Helped |
| Replace standard patch embedding with multi-scale adaptive kernels | ViTDet (ViT-B) · COCO2017, 256 | box AP: 0.19 -> 0.30 | Helped |
| Use MSPE rather than image resizing at inference | ViT-B · ImageNet-1K, 28×28 | Top-1 accuracy: 52.99 -> 56.41 | Helped |
| Increase training from 1 to 5 epochs (ablation) | MSPE ViT-B · ImageNet-1K, 28×28 | Top-1 accuracy: 28.07 -> 56.42 | Helped |
| Increase training from 5 to 20 epochs (ablation) | MSPE ViT-B · ImageNet-1K, 28×28 | Top-1 accuracy: 56.42 -> 58.88 | Helped |
| Set λ to 0 rather than 1 or 2 (ablation) | MSPE · ImageNet-1K | Top-1 accuracy | Hurt |
| Increase kernel count from K=3 to K=4 (ablation) | MSPE · ImageNet-1K | Top-1 accuracy | No clear effect |
| Use PI-resize for resizing patch embedding weights instead of other resizing methods (ablation) | MSPE · ImageNet-1K | Top-1 accuracy | Helped |
Surprising findings
- Higher patch-embedding similarity to the class token does not guarantee better accuracy; MSPE achieves higher accuracy than FlexiViT despite lower patch-embedding similarity.
- Vanilla ViT performance at 448x448 is lower than at 224x224 for ViT-B (53.81 vs 85.10), while FlexiViT stays stable.
Limitations
- Position embedding is not studied; bilinear interpolation of position embedding is simply adopted.
- Encoder is frozen, so the approach cannot jointly adapt the transformer; the paper notes joint optimization as future work.
- Performance at high resolution (448) is only comparable to FlexiViT, not clearly superior.
Experimental setup
Datasets: ImageNet-1K ADE20K Cityscapes COCO2017
Benchmarks: ImageNet-1K ADE20K Cityscapes COCO2017
Metrics: top-1 accuracy mIOU mACC F1-score AP
Baselines: Vanilla ViT FlexiViT ResFormer NaViT IMG-resize
Builds on: FlexiViT ResFormer NaViT ViT PI-resize
Closest papers by meaning
- Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More (ICML 2025 · similarity 0.87)
- Win-Win: Training High-Resolution Vision Transformers from Two Windows (ICLR 2024 · similarity 0.85)
- No Pains, More Gains: Recycling Sub-Salient Patches for Efficient High-Resolution Image Recognition (CVPR 2025 · similarity 0.85)
- InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD (NeurIPS 2024 · similarity 0.84)
- Charm: The Missing Piece in ViT Fine-Tuning for Image Aesthetic Assessment (CVPR 2025 · similarity 0.84)
- PatchVSR: Breaking Video Diffusion Resolution Limits with Patch-wise Video Super-Resolution (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.