Research map / Architectures and efficiency / Attention and transformers
Win-Win: Training High-Resolution Vision Transformers from Two Windows
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Win-Win trains non-hierarchical ViTs on high-resolution dense tasks by keeping only N=2 random square windows of tokens per training image (with RoPE and an attention-temperature correction), enabling single-pass full-resolution inference for segmentation, depth, and optical flow.
Problem
Training vanilla vision transformers for high-resolution pixelwise prediction is prohibitively costly, and models trained on low-resolution crops generalize poorly when tested at higher resolutions, typically requiring slow tiling or resizing.
Approach
Mask most high-resolution tokens during training, keeping only a structured set of N=2 random non-overlapping square windows so that the model learns both local (intra-window) and global (inter-window) interactions; combined with RoPE and a test-time softmax temperature correction, the model runs directly on full-resolution inputs. For optical flow, the second frame's windows are sampled guided by
Key results
- Two 22x22 token windows (968 tokens) reach 63.6 mIoU on BDD segmentation, versus 63.2 for full-resolution ViT, at about 4x faster training and 2x lower memory.
- On Spring (Full-HD), Win-Win reaches EPE 0.475 and s40+ 3.639, beating CroCo-Flow (0.498 EPE, s40+ 4.046) while being over an order of magnitude faster at inference.
- On MPI-Sintel clean test, Win-Win gets EPE 1.15 with a single forward pass, competitive with tiling-based transformer methods such as FlowFormer++ (1.07).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with 2 windows instead of a single window (ablation) | Win-Win semantic segmentation · BDD-100k validation | mIoU: 61.8 -> 63.6 | Helped |
| Use 3 windows instead of 2 (ablation) | Win-Win semantic segmentation · BDD-100k validation | mIoU: 63.6 -> 63.2 | Hurt |
| Use 6 windows instead of 2 (ablation) | Win-Win semantic segmentation · BDD-100k validation | mIoU: 63.6 -> 62.5 | Hurt |
| Add a softmax temperature (ablation) | Win-Win semantic segmentation with 2 windows · BDD-100k validation | mIoU: 63.6 -> 63.6 | No clear effect |
| Replace RoPE with cosine absolute positional embeddings (ablation) | Win-Win semantic segmentation · BDD-100k validation | mIoU: 63.6 -> 57.0 | Hurt |
| Use unequal window sizes of 28×28 and 15×15 (ablation) | Win-Win semantic segmentation · BDD-100k validation | mIoU: 63.6 -> 63.7 | No clear effect |
| Reduce the second window to 12×12 (ablation) | Win-Win semantic segmentation · BDD-100k validation | mIoU: 63.6 -> 62.8 | Hurt |
| Add isolated random tokens to the two-window setup (ablation) | Win-Win semantic segmentation · BDD-100k validation | mIoU: 63.6 -> 63.3; 63.6 -> 63.2; 63.6 -> 63.0 | Hurt |
| Randomize window count, token count, token ratio, and aspect ratio (ablation) | Win-Win semantic segmentation · BDD-100k validation | mIoU: 63.6 -> 63.5; 63.6 -> 63.6 | No clear effect |
| Increase frame-2 window count from 2 to 4 (ablation) | Flow-guided Win-Win optical flow · Synthetic optical flow | EPE: 1.92 -> 1.17 | Helped |
| Set frame-2 window sampling stochasticity to 0.3 (ablation) | Flow-guided Win-Win optical flow · Synthetic optical flow | EPE: 1.17 -> 0.96 | Helped |
| Use the same windows in both frames (ablation) | Win-Win optical flow · Synthetic optical flow | EPE: 1.17 -> 2.70 | Hurt |
Surprising findings
- Using two windows performs best, and additional isolated random tokens tend to degrade performance rather than help.
- Random uniform token masking (MAE-style) fails to generalize to full resolution for optical flow, with poor convergence.
Limitations
- Performance still drops at lower test resolutions relative to models trained at the target resolution.
- Optical flow results on MPI-Sintel and Spring trail the best task-specific methods on some metrics (e.g., GMFlow+ on clean Sintel, CroCo-Flow on 1px outliers), and the flow model uses a ViT-Base backbone versus ViT-Large for CroCo-Flow.
- Temperature correction did not help with absolute positional embeddings, so the method depends on relative positional embeddings such as RoPE.
Experimental setup
Datasets: BDD100k BDD10k FlyingChairs FlyingThings MPI-Sintel TartanAir Spring AutoFlow-style synthetic flow dataset
Benchmarks: BDD10k semantic segmentation MPI-Sintel Spring
Metrics: mIoU EPE 1px outlier rate s0-10/s10-40/s40+ EPE
Baselines: Full-res ViT ViT-Det crop training with tiling CroCoFlow FlowFormer FlowFormer++ GMFlow+ SKFlow MS-Raft+ RAFT CRAFT PWC-Net+
Builds on: CroCo CroCo-Flow ViT-Det RoPE DPT
Closest papers by meaning
- Taming Transformers for High-Resolution Image Synthesis (CVPR 2021 · similarity 0.90)
- Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuning (NeurIPS 2022 · similarity 0.88)
- VITRIX-UniViTAR: Unified Vision Transformer with Native Resolution (NeurIPS 2025 · similarity 0.88)
- An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels (ICLR 2025 · similarity 0.87)
- Scaling White-Box Transformers for Vision (NeurIPS 2024 · similarity 0.87)
- Vision Transformers with Self-Distilled Registers (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.