Research map / Architectures and efficiency / Attention and transformers
Sub-token ViT Embedding via Stochastic Resonance Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Stochastic Resonance Transformer (SRT), a training-free test-time method that shifts input images by sub-token pixel translations, runs a frozen ViT, and aggregates inverse-aligned features to obtain sub-token-resolution embeddings that improve dense and some non-dense vision tasks.
Problem
Vision Transformer features are spatially coarse because each token covers a non-overlapping rectangular patch, losing fine spatial detail needed for fine-grained dense prediction tasks.
Approach
Apply deterministic sub-token translations to the input image, extract ViT features for each shifted copy, undo the shift on the feature maps, and average them, borrowing the stochastic resonance principle from signal processing to recover sub-patch spatial resolution without modifying weights or the forward pass.
Key results
- SRT improves DINOV2-ViT-S/14 linear-head depth RMSE_log by 14.9% relative (0.125 to 0.108 range reported in table as 0.162 to 0.140 for ViT-S/14 linear).
- On DAVIS-2017, SRT raises DINO ViT-S/16 F&J from 0.617 to 0.642 (about 4.1% relative), average 2.4% across three DINO backbones.
- On TokenCut unsupervised salient segmentation, SRT improves maxF by an average of 1.8%, while naive color-jitter ensemble degrades DAVIS F&J to 0.477.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply SRT feature-level ensembling | DINO ViT-S/16 · DAVIS-2017 | F&J: 0.617 -> 0.642 | Helped |
| Apply SRT feature-level ensembling | DINO ViT-B/16 · DAVIS-2017 | F&J: 0.622 -> 0.630 | Helped |
| Apply SRT feature-level ensembling | DINO ViT-S/8 · DAVIS-2017 | F&J: 0.706 -> 0.720 | Helped |
| Apply SRT to backbone features | DINOv2 ViT-B/14 Linear · NYU-V2 | RMSE: 0.396 -> 0.349 | Helped |
| Apply SRT to backbone features | DINOv2 ViT-B/14 Linear · NYU-V2 | RMSE_log: 0.135 -> 0.108 | Helped |
| Apply SRT to backbone features | DINOv2 ViT-B/14 DPT · NYU-V2 | RMSE: 0.323 -> 0.305 | Helped |
| Aggregate model outputs instead of intermediate features (ablation) | DINOv2 ViT-B/14 DPT · NYU-V2 | RMSE: 0.323 -> 0.314 | Hurt |
| Apply SRT feature-level ensembling | DINO ViT-S/16 · ECSSD | maxF: 80.3 -> 82.4 | Helped |
| Distill SRT representations into the ViT (ablation) | DINO ViT-S/16 · DAVIS-2017 | F&J: 0.617 -> 0.625 | Helped |
| Use overlapping tokens at inference (ablation) | DINO ViT-S/16 · DAVIS-2017 | F&J: 0.617 -> 0.591 | Hurt |
| Replace SRT perturbations with color jitter and feature-level ensembling (ablation) | DINO ViT-S/16 · DAVIS-2017 | F&J: 0.617 -> 0.477 | Hurt |
| Replace SRT with bilinear feature interpolation (ablation) | DINOv2 ViT-S/14 DPT · NYU-V2 | RMSE: 0.336 -> 0.573 | Hurt |
Surprising findings
- Naive output-space ensembling (OE) and naive color-jitter feature ensembling often hurt performance, while feature-level SRT helps.
- Interpolating features up to full resolution (bilinear/bicubic) substantially degrades depth prediction versus the baseline.
Limitations
- Inference cost and latency increase linearly with the number of perturbed forward passes.
- Gains on TokenCut-based segmentation are constrained by the coarse token-level graph construction; full-resolution features are too costly for the O(n^2) graph cut.
- Larger perturbation levels cause information loss at image boundaries, so performance gains trade off against perturbation magnitude.
Experimental setup
Datasets: DAVIS-2017 NYU-V2 ECSSD DUTS DUT-OMRON Oxford image retrieval VOC07 ADE20K CIFAR
Benchmarks: DAVIS-2017 NYU-V2 ECSSD DUTS DUT-OMRON Oxford image retrieval VOC07 ADE20K CIFAR
Metrics: F&J score RMSE RMSE_log AbsRel maxF IoU mAP detection rate mIoU accuracy
Baselines: Naive ensemble Overlapping tokens Output-space ensemble Bilinear interpolation feature upsampling Bicubic interpolation feature upsampling
Builds on: Stochastic Resonance TokenCut DINO DINOv2 DPT
Closest papers by meaning
- Channel Vision Transformers: An Image Is Worth 1 x 16 x 16 Words (ICLR 2024 · similarity 0.86)
- Your ViT is Secretly an Image Segmentation Model (CVPR 2025 · similarity 0.86)
- Slicing Vision Transformer for Flexible Inference (NeurIPS 2024 · similarity 0.86)
- SATA: Spatial Autocorrelation Token Analysis for Enhancing the Robustness of Vision Transformers (CVPR 2025 · similarity 0.85)
- Bi-ViT: Pushing the Limit of Vision Transformer Quantization (AAAI 2024 · similarity 0.85)
- GradQ-ViT: Robust and Efficient Gradient Quantization for Vision Transformers (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.