Research map / Architectures and efficiency / Attention and transformers
NuWa: Deriving Lightweight Class-Specific Vision Transformers for Edge Devices
Ziteng Wei, Qiang He, Bing Li, Feifei Chen, Hai Jin, Yun Yang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add self-knowledge purification (SKP) (ablation) | NuWa · ImageNet S4-S6 (25 classes), alpha=0.6 | avg accuracy: 72.69 -> 91.44 | Improved |
| add MHA pruning (balanced with MLP via rho) (ablation) | NuWa · ImageNet S4-S6 (25 classes), alpha=0.6 | avg accuracy: 76.05 -> 91.44 | Improved |
| prune MLP in OFP rather than only MHA (ablation) | NuWa · ImageNet S4-S6 (25 classes), alpha=0.6 | avg accuracy: 5.49 -> 91.44 | Improved |
| select retained MLP neurons by activation instead of randomly (ablation) | NuWa · ImageNet S4-S6 (25 classes), alpha=0.6 | avg accuracy: 86.45 -> 91.44 | Improved |
| use closed-form solution for MLP W2 instead of direct removal (ablation) | NuWa · ImageNet S4-S6 (25 classes), alpha=0.6 | avg accuracy: 76.69 -> 91.44 | Improved |
| add light fine-tuning (10 epochs) to NuWa | NuWa · ImageNet S4-S6 (25 classes), alpha=0.6 | avg accuracy: 91.44 -> 95.95 | Improved |
| NuWa vs training-dependent baselines without retraining | DeiT-Base pruning · ImageNet S7-S9 (50 classes), alpha=0.6 | avg accuracy: 89.50 -> 85.83 (vs X-Pruner) | Hurt |
| smaller SKP batch size (ablation) | NuWa SKP · ImageNet S4-S6 | anchor pruning rate | Improved |
| replace training-dependent pruning with NuWa for derivation cost | class-specific ViT derivation · ImageNet S1-S9, N=50, M=10 | AWS cost: $23074 -> $59.12 vs X-Pruner | Improved |
| SKP on DeiT-Base anchor model vs unpruned | DeiT-Base · ImageNet S1-S3 (10 classes) | avg accuracy: 78.60 -> 97.67 | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- X-Pruner — X-Pruner: eXplainable Pruning for Vision Transformers (CVPR 2023) · better on 3 of 5 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Vulcan: Crafting Compact Class-Specific Vision Transformers For Edge Intelligence (ICLR 2026 · similarity 0.92)
- DeepCompress-ViT: Rethinking Model Compression to Enhance Efficiency of Vision Transformers at the Edge (CVPR 2025 · similarity 0.90)
- Rethinking Depth Pruning for Vision Transformers: A Heterogeneity-Aware Perspective (ICML 2026 · similarity 0.89)
- Width & Depth Pruning for Vision Transformers (AAAI 2022 · similarity 0.88)
- Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformers (CVPR 2024 · similarity 0.87)
- Linearly Decomposing and Recomposing Vision Transformers for Diverse-Scale Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
