Research map / Architectures and efficiency / Attention and transformers

Patch Rebirth: Fast and Transferable Model Inversion of Vision Transformers

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Efficient and robust vision transformers · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Patch Rebirth Inversion (PRI), which detaches top-attention patches at successive points within one ViT model-inversion run instead of pruning low-attention patches, yielding faster inversion and more transferable synthetic data than SMI and DMI.

Problem

Model inversion of Vision Transformers is computationally expensive due to self-attention over all patches, and prior sparse model inversion (SMI) prunes low-importance patches, which the authors argue discards transferable knowledge and limits diversity.

Approach

Rather than permanently pruning unimportant patches, PRI detaches the most important patches at scheduled points during a single inversion run to form multiple sparse images, letting remaining patches continue evolving ("Re-Birth" effect) so they later accumulate class-relevant and class-agnostic knowledge.

Key results

• PRI at 50% sparsity on DeiT-Base achieves 2.88 its/s vs 2.20 for SMI (+30.9%) and at 86% sparsity reaches 11.81 its/s vs 5.58 for SMI (+111.6%) on DeiT-Base, versus 1.10 for DMI.
• On CIFAR-100 knowledge distillation into DeiT-Base, PRI at 50% sparsity reaches 80.05% vs 79.83% for DMI and 79.52% for SMI.
• On ImageNet-1k W4/A8 quantization with DeiT-Base, PRI at 50% sparsity reaches 80.30% vs 80.19% for DMI and 80.18% for SMI.

Surprising findings

• Randomly or low-attention-selected patches achieve nearly the same downstream performance as high-attention patches when inversion continues, suggesting patch selection criterion matters little.
• Students trained on single-class (airplane) PRI-inverted images generalize reasonably well to all 10 CIFAR-10 classes, while DMI and SMI do not.

Limitations

• Uses a simple uniform detachment schedule; adaptive detachment policies are not explored.
• For DeiT-Tiny students at high sparsity (86%), DMI clearly outperforms PRI and SMI.
• Efficiency numbers are reported only on a single RTX A6000 GPU.

Experimental setup

Datasets: ImageNet-1k CIFAR-10 CIFAR-100 Tiny-ImageNet

Benchmarks: ImageNet-1k CIFAR-10 CIFAR-100 Tiny-ImageNet

Metrics: throughput (iterations/s) FLOPs GPU memory top-1 accuracy KL divergence

Baselines: DMI SMI PSAQ-ViT MimiQ

Builds on: Sparse Model Inversion Dense Model Inversion token pruning learned step size quantization

Closest papers by meaning

• Sparse Model Inversion: Efficient Inversion of Vision Transformers for Data-Free Applications (ICML 2024 · similarity 0.92)
• Model Inversion with Layer-Specific Modeling and Alignment for Data-Free Continual Learning (NeurIPS 2025 · similarity 0.87)
• Sparse autoencoders reveal selective remapping of visual concepts during adaptation (ICLR 2025 · similarity 0.87)
• ModHiFi: Identifying High Fidelity predictive components for Model Modification (NeurIPS 2025 · similarity 0.87)
• Linearly Decomposing and Recomposing Vision Transformers for Diverse-Scale Models (NeurIPS 2024 · similarity 0.86)
• Split Adaptation for Pre-trained Vision Transformers (CVPR 2025 · similarity 0.86)