Research map / Architectures and efficiency / Pruning and efficient tuning
Sparse Model Inversion: Efficient Inversion of Vision Transformers for Data-Free Applications
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Sparse Model Inversion (SMI), a plug-and-play strategy that progressively stops inverting background patches in ViTs using last-layer CLS attention, speeding up dense inversion up to 3.79x while keeping or improving data-free quantization and knowledge transfer performance.
Problem
Dense model inversion reconstructs the entire image, which is extremely slow for high-resolution inputs from large-scale ViTs, and it also inverts noisy backgrounds and spurious foreground-background correlations.
Approach
Use the CLS-token attention from the previous iteration to identify semantic patches and progressively discard low-attention background patches from both forward and backward passes during inversion, as a plug-and-play addition that leaves the dense loss unchanged.
Key results
- Sparse DeepInversion on DeiT/16-Base reaches 4.51 its/s vs 1.19 dense (x3.79) and W4/A8 quantized top-1 of 77.51 vs 75.99 for dense DeepInversion.
- On CIFAR-10 knowledge transfer with DeiT/16-Tiny, sparse inversion gives 90.08 top-1 vs 69.51 for dense DeepInversion.
- For DeiT/16-Base knowledge transfer on CIFAR-10, sparse data needs 5540 samples and 43 iterations vs 14080 samples and 110 iterations for dense data at equal accuracy (90.02).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use sparse rather than dense inversion | DeiT/16-Tiny model quantization · ImageNet validation, W4/A8 | Top-1: 64.28 -> 64.04 | Hurt |
| Use sparse rather than dense inversion | DeiT/16-Tiny model quantization · ImageNet validation, W8/A8 | Top-1: 70.27 -> 70.13 | Hurt |
| Use sparse rather than dense inversion | DeiT/16-Base model quantization · ImageNet validation, W4/A8 | Top-1: 75.99 -> 77.51 | Helped |
| Use sparse rather than dense inversion | DeiT/16-Base model quantization · ImageNet validation, W8/A8 | Top-1: 78.58 -> 79.63 | Helped |
| Use sparse rather than dense inversion | DeiT/16-Tiny and DeiT/16-Base inversion · ImageNet inversion | Throughput (its/s): 7.33 -> 18.82; 1.19 -> 4.51 | Helped |
| Use sparse rather than dense inversion | DeiT/16-Tiny and DeiT/16-Base knowledge transfer · CIFAR-10 | Top-1: 69.51 -> 90.08; 90.02 -> 95.10 | Helped |
| Use sparse rather than dense inversion | DeiT/16-Tiny and DeiT/16-Base knowledge transfer · CIFAR-100 | Top-1: 70.32 -> 70.48; 74.88 -> 74.53 | Mixed |
| Use multi-stage rather than one-stage stopping at the same sparsity (ablation) | DeiT/16-Tiny knowledge transfer · CIFAR-10 | Data-free knowledge transfer: 88.82 -> 90.08 | Helped |
| Vary the stopping schedule while keeping overall sparsity at 77% (ablation) | DeiT/16-Tiny knowledge transfer · CIFAR-10 | Data-free knowledge transfer: 88.82; 89.02; 89.13; 89.26; 89.38; 90.02; 89.90; 90.08; 90.14 | No clear effect |
| Increase batch size during sparse inversion (ablation) | DeiT/16-Base · Model inversion | Speed gains: ×3.21; ×3.52; ×3.79 | Helped |
| Increase model scale during sparse inversion (ablation) | DeiT/16-Tiny versus DeiT/16-Base · Model inversion | Speed gains: ×2.57; ×3.79 | Helped |
| Use sparse rather than dense inverted training data | Knowledge transfer from DeiT/16-Base · CIFAR-10 convergence conditions | Sample complexity (N): 14080 -> 5540 | Helped |
Surprising findings
- Sparsely inverted data sometimes improves downstream accuracy over dense inversion, e.g., quantization W4/A8 on DeiT-Base is higher with sparse data (77.51 vs 75.99).
- Dense inversion for DeiT/16-Tiny knowledge transfer on CIFAR-10 caused near training failure (69.51), which sparse inversion avoided.
Limitations
- Evaluated only on ViT-based classifiers (DeiT, CLIP ViT) and small-scale classification downstream tasks.
- Relies on the assumption that last-layer CLS attention reflects label-relevant patches, with theory built on a simplified convergence framework.
Experimental setup
Datasets: ImageNet CIFAR-10 CIFAR-100 MiniImageNet Oxford 102 Flower CUB-200-2011
Benchmarks: ImageNet CIFAR-10 CIFAR-100
Metrics: throughput (its/s) FLOPs (G) GPU memory (MB) top-1 accuracy
Baselines: DeepInversion PSAQ-ViT Gaussian Noise calibration
Builds on: DeepInversion PSAQ-ViT token sparsification
Closest papers by meaning
- Patch Rebirth: Fast and Transferable Model Inversion of Vision Transformers (NeurIPS 2026 · similarity 0.92)
- Semantic Image Inversion and Editing using Rectified Stochastic Differential Equations (ICLR 2025 · similarity 0.87)
- Gradient-free Decoder Inversion in Latent Diffusion Models (NeurIPS 2024 · similarity 0.86)
- Generative Densification: Learning to Densify Gaussians for High-Fidelity Generalizable 3D Reconstruction (CVPR 2025 · similarity 0.86)
- Model Inversion with Layer-Specific Modeling and Alignment for Data-Free Continual Learning (NeurIPS 2025 · similarity 0.86)
- SCube: Instant Large-Scale Scene Reconstruction using VoxSplats (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.