AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Revisiting Vision Transformer from the View of Path Ensemble

Shuning Chang, Pichao Wang, Hao Luo, Fan Wang, Mike Zheng Shou

ICCV 2023 · Architectures and efficiency › Attention and transformers · approach: Hybrid and interpretable ViT designs · 10 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
prune short paths (keep p7-p12) in the ensemble prediction (ablation)DeiT-S · ImageNet-1K valtop-1 accuracy (%): 79.8 -> 80.2Improved
add EnsembleScale to re-weight paths (ablation)DeiT-S · ImageNet-1K valtop-1 accuracy (%): 79.8 -> 80.3Improved
add EnsembleScale to 18-layer DeiT-S (ablation)DeiT-S (18 layers) · ImageNet-1K valtop-1 accuracy (%): 80.1 -> 81.2Improved
add EnsembleScale to 24-layer DeiT-S (ablation)DeiT-S (24 layers) · ImageNet-1K valtop-1 accuracy (%): 78.9 -> 81.6Improved
apply EnsembleScale and path pruning to Swin-T (ablation)Swin-T · ImageNet-1K valtop-1 accuracy (%): 81.3 -> 81.5Improved
add dynamic early exit using first seven pathsDeiT-S · ImageNet-1K valFLOPs and accuracy: 4.58G -> 3.42G, accuracy 79.8 -> 79.8Improved
add EnsembleScale before fine-tuning on CIFAR (ablation)DeiT-S · CIFAR-100accuracy (%): 85.9 -> 87.0Improved
add prediction-logit then hidden-state self-distillation between paths (ablation)DeiT-S · ImageNet-1K valtop-1 accuracy (%): 79.8 -> 80.2 -> 80.6Improved
combine self-distillation with path combination (ablation)DeiT-S · ImageNet-1K valtop-1 accuracy (%): 80.6 -> 81.0Improved
add hidden-state distillation and path combination to Swin-B prediction-logit distillation (ablation)Swin-B · ImageNet-1K valtop-1 accuracy (%): 84.3 -> 84.2Hurt
use a larger teacher-student distance (Delta=4 vs 2) (ablation)self-distillation on DeiT-S · ImageNet-1K valtop-1 accuracy (%): 81.0 -> 80.4Hurt
include very short paths in distillation (start from p0 instead of p5) (ablation)self-distillation on DeiT-S · ImageNet-1K valtop-1 accuracy (%): 81.0 -> 80.6Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy