Research map / Architectures and efficiency / Attention and transformers
MPCViT: Searching for Accurate and Efficient MPC-Friendly Vision Transformer with Heterogeneous Attention
Wenxuan Zeng, Meng Li, Wenjie Xiong, Tong Tong, Wen-jie Lu, Jin Tan, Runsheng Wang, Ru Huang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Softmax attention with ReLU Softmax (ablation) | ViT attention in MPC · CIFAR-10 | top-1 accuracy (%) / latency (s): 6.82 s -> 5.32 s; 92.69 -> 92.31 | Improved |
| replace all Softmax with Scaling attention (ablation) | ViT attention in MPC · CIFAR-10 | top-1 accuracy (%): 92.69 -> 91.52; 6.82 s -> 0.66 s | Mixed |
| use Linformer low-rank attention (ablation) | ViT attention in MPC · CIFAR-10 | top-1 accuracy (%): 92.69 -> 90.85 | Hurt |
| use head-wise rather than layer-wise or row-wise heterogeneous attention search (ablation) | MPCViT · CIFAR-10 (mu=0.5) | top-1 accuracy (%): 93.01 / 93.16 -> 93.21 | Improved |
| add logits-based knowledge distillation (ablation) | MPCViT · Tiny-ImageNet (mu=0.7) | top-1 accuracy (%): 59.02 -> 62.39 | Improved |
| add feature-based KD on top of logits KD (ablation) | MPCViT · Tiny-ImageNet (mu=0.7) | top-1 accuracy (%): 62.39 -> 63.03 | Improved |
| add additional ReLU after fused linear layer in MPCViT+ (ablation) | MPCViT+ · CIFAR-10 (mu=0.7) | top-1 accuracy (%): 93.81 -> 94.27 | Improved |
| linearize GeLU in MLP (MPCViT+) | MPCViT · CIFAR-10/100 | top-1 accuracy (%) | Mixed |
| search per-head replacement instead of per-layer uniform replacement (ablation) | MPCViT · CIFAR-10/100, Tiny-ImageNet | accuracy-latency Pareto front | Improved |
| replace attention heads with ScaleAttn instead of pruning heads (ablation) | MPCViT · CIFAR-10 | top-1 accuracy (%): 93.03 -> 93.38 | Improved |
| mix Softmax attention with ScaleAttn instead of ReLU Softmax (ablation) | MPCViT · CIFAR-10/100, Tiny-ImageNet | latency: at least 1.4x and 1.2x latency | Hurt |
Closest papers by meaning
- SAL-ViT: Towards Latency Efficient Private Inference on ViT using Selective Attention Search with a Learnable Softmax Approximation (ICCV 2023 · similarity 0.90)
- EfficientViT: Memory Efficient Vision Transformer With Cascaded Group Attention (CVPR 2023 · similarity 0.88)
- FasterViT: Fast Vision Transformers with Hierarchical Attention (ICLR 2024 · similarity 0.88)
- Bi-ViT: Pushing the Limit of Vision Transformer Quantization (AAAI 2024 · similarity 0.87)
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.86)
- DeepCompress-ViT: Rethinking Model Compression to Enhance Efficiency of Vision Transformers at the Edge (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
