Research map / Architectures and efficiency / Attention and transformers

Configuring Data Augmentations to Reduce Variance Shift in Positional Embedding of Vision Transformers

AAAI 2025 · Architectures and efficiency › Attention and transformers · approach: Vision transformer design · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that Mixup and upsampling of positional embeddings shift variance in ViTs at test time, and proposes rescaling upsampled positional embedding by 1/sqrt(k) plus configuring random erasing (pixel mode, default mean-std) to remove the shift.

Problem

Rich data augmentations such as Mixup and random erasing, together with bicubic upsampling of positional embeddings at test time, cause variance shifts in ViT positional embeddings between training and test, degrading test performance.

Approach

Analyze the variance of patch and positional embeddings after LayerNorm; show interpolation-type upsampling scales variance by k, Mixup scales variance by lambda^2+(1-lambda)^2, while Cutmix and pixel-mode random erasing (with default mean-std) conserve variance; fix by rescaling upsampled positional embedding by 1/sqrt(k) and configuring augmentations accordingly.

Key results

• Rescaling upsampled positional embedding by 1/sqrt(k) improves ViT-B/16 top-1 accuracy at 864^2 test size from 61.33% to 67.28% (+5.95), while slightly degrading the 224^2 size (-0.18).
• On ImageNet training, Mixup 0.4 with Cutmix reduces ViT-B/16 top-1 from 80.240% (Cutmix only) to 79.248%, supporting that Mixup hurts ViT training.
• Rescaling improves ADE20K mIoU from 43.16 to 43.27 and LoveDA mIoU from 47.78 to 48.37 with DeiT-S/16 + UPerNet.

Measured effects reported in the paper

Surprising findings

• Mixup, a standard ViT training augmentation, reduced ImageNet top-1 accuracy when combined with Cutmix, versus Cutmix alone.
• Nearest-neighbor upsampling conserves variance but performed poorly, so the authors instead keep bicubic upsampling and rescale.

Limitations

• Analysis assumes mean-zero positional embedding and large vectors, with variance shift framed under mean consistency.
• Nearest-neighbor upsampling, which conserves variance, performs poorly, so the fix relies on a heuristic 1/sqrt(k) rescaling.
• Experiments are on a limited set of ViT sizes and segmentation datasets; RandAugment/AutoAugment effects are only discussed conjecturally.

Experimental setup

Datasets: ImageNet-1K ImageNet-21K ADE20K LoveDA Cityscapes

Benchmarks: ImageNet-1K ADE20K LoveDA Cityscapes

Metrics: top-1 accuracy aAcc mIoU mAcc

Baselines: bicubic upsampling of positional embedding (baseline) Mixup inception-mean-std

Builds on: Vision Transformer Mixup CutMix random erasing DeiT

Closest papers by meaning

• Making Vision Transformers Truly Shift-Equivariant (CVPR 2024 · similarity 0.86)
• Accelerating Augmentation Invariance Pretraining (NeurIPS 2024 · similarity 0.86)
• Linearly Decomposing and Recomposing Vision Transformers for Diverse-Scale Models (NeurIPS 2024 · similarity 0.86)
• Vision Transformers with Self-Distilled Registers (NeurIPS 2025 · similarity 0.85)
• Vision Transformer Neural Architecture Search for Out-of-Distribution Generalization: Benchmark and Insights (NeurIPS 2024 · similarity 0.85)
• Charm: The Missing Piece in ViT Fine-Tuning for Image Aesthetic Assessment (CVPR 2025 · similarity 0.85)