Research map / Architectures and efficiency / Attention and transformers
Real-time Core-Periphery Guided ViT with Smart Data Layout Selection on Mobile Devices
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ECP-ViT, a core-periphery guided sparse self-attention for ViTs combined with compiler-level data layout selection that eliminates Reshape/Transpose overhead, enabling real-time ViT inference on mobile GPUs.
Problem
Efficient ViT variants that reduce FLOPs (local attention, token pruning) incur data transformation overheads (Reshape, Transpose) and irregular memory access, so they are often slower than vanilla ViT on memory-bandwidth-limited mobile GPUs.
Approach
Replace full self-attention with a core-periphery graph structure (core patches attend to all, periphery patches attend only to cores, guided by Grad-CAM), and co-design a compiler that chooses intermediate data layouts so that Transpose/Reshape/Slice operators are eliminated at kernel level.
Key results
- ECP-ViT-Base reaches 84.6% top-1 on ImageNet-1K with 86.5M params and 16.96G MACs, vs. 83.9% for ViT-Base/16.
- End-to-end on OnePlus 11 mobile GPU, ECP-ViT-tiny runs in 15.2 ms vs. 115.3 ms for MNN on ViT-tiny baseline in the paper's table, with reported speedups of 4.6x to 26.9x over other frameworks.
- ECP-ViT-S/16 improves STL-10 accuracy from 96.36% to 98.18% and CIFAR-100 from 89.51% to 89.56%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use core-periphery guided self-attention at the reported best core ratio | ECP-ViT-Base · ImageNet-1K | Top-1 accuracy: 83.9 -> 84.6 | Helped |
| Use core-periphery guided self-attention | ViT-S/16 · STL-10 | Top-1 accuracy: +1.82 | Helped |
| Use core-periphery guided self-attention | ViT-S/16 · CIFAR-100 | Top-1 accuracy: +0.05 | Helped |
| Use core-periphery guided self-attention | ViT-S · ImageNet-1K | Top-1 accuracy: 81.1 -> 80.9 | Hurt |
| Use core-periphery guided self-attention | ViT-Tiny · ImageNet-1K | Top-1 accuracy: 75.1 -> 75.8 | Helped |
| Use core ratio 0.9 instead of 1.0 (ablation) | ECP-ViT-B/16 · ImageNet-1K | Top-1 accuracy: 83.89 -> 84.62 | Helped |
| Use core ratio 0.8 instead of 1.0 (ablation) | ECP-ViT-B/16 · ImageNet-1K | Top-1 accuracy: 83.89 -> 84.51 | Helped |
| Vary core ratio below 1.0 (ablation) | ECP-ViT-T/16 · ImageNet-1K | Top-1 accuracy: 75.90 -> 65.50–75.84 | Hurt |
| Vary core ratio below 1.0 (ablation) | ECP-ViT-S/16 · ImageNet-1K | Top-1 accuracy: 81.12 -> 73.24–80.88 | Hurt |
| Use core ratios 0.1–0.7 instead of 1.0 (ablation) | ECP-ViT-B/16 · ImageNet-1K | Top-1 accuracy: 83.89 -> 76.15–84.03 | Hurt |
| Apply ECP-ViT compiler optimizations to vanilla ViT inference | ViT-Tiny, ViT-Small, ViT-Base · ImageNet-1K on OnePlus 11 GPU | Latency speedup: 4.8×; 4.9×; 5.3× | Helped |
| Apply ECP-ViT compiler optimizations to core-periphery ViT inference | ECP-ViT-Tiny, ECP-ViT-Small, ECP-ViT-Base · ImageNet-1K on OnePlus 11 GPU | Latency speedup versus other frameworks: 4.6× to 26.9× | Mixed |
Surprising findings
- At a core ratio of 0.9, ECP-ViT-Base surpasses the full-attention ViT-Base baseline, suggesting sparsity of self-attention can improve accuracy.
- Swin-T, despite fewer FLOPs, is over 3x slower than ViT-T on a mobile GPU because local attention adds layout transformation overhead.
Limitations
- Evaluation restricted to mobile GPUs; NPU and CPU backends not explored.
- Accuracy gains rely on fine-tuning from ImageNet-21K pretrained ViT weights.
- TVM and other frameworks lack systematic optimizations, so speedup comparisons depend on framework support.
Experimental setup
Datasets: ImageNet-21K ImageNet-1K STL-10 CIFAR-100 TinyImageNet
Benchmarks: ImageNet-1K STL-10 CIFAR-100 TinyImageNet
Metrics: top-1 accuracy latency (ms) speedup peak memory cache miss rate MACs parameters
Baselines: ViT-Base DeiT-Base Swin MetaFormer PVT TNT ConViT MNN TNN TVM
Builds on: Vision Transformer (ViT) Grad-CAM MNN DNNFusion
Closest papers by meaning
- SHViT: Single-Head Vision Transformer with Memory Efficient Macro Design (CVPR 2024 · similarity 0.87)
- RepViT: Revisiting Mobile CNN From ViT Perspective (CVPR 2024 · similarity 0.87)
- Mobile Attention: Mobile-Friendly Linear-Attention for Vision Transformers (ICML 2024 · similarity 0.86)
- HydraViT: Stacking Heads for a Scalable ViT (NeurIPS 2024 · similarity 0.86)
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.86)
- FasterViT: Fast Vision Transformers with Hierarchical Attention (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.