Research map / Architectures and efficiency / Pruning and efficient tuning
ModHiFi: Identifying High Fidelity predictive components for Model Modification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ModHiFi, a data-free, loss-free model modification method that scores components by singleton Subset Fidelity (from synthetic data) to perform structured pruning and classwise unlearning on CNNs, ViTs/Swin, and LLMs.
Problem
Modifying well-trained open-weight models (structured pruning and classwise unlearning) without training data, the loss function, or gradients, using only distributional access such as synthetic data.
Approach
Defines Subset Fidelity, the fraction of a layer's output a subset of input contributions can reconstruct after optimal weight compensation; proves global error is linearly bounded by local reconstruction error for Lipschitz networks, and that top-k singleton fidelity is optimal when input contributions are uncorrelated.
Key results
- ModHiFi-P on ResNet50/ImageNet reaches 73.82% accuracy at 3.66x FLOP reduction and 2.42x CPU speedup, versus 73.80% for DFPC at 3.46x FLOP and 2.37x CPU speedup.
- On Llama-2-7B at 10% sparsity, ModHiFi-P-Alpaca reaches WikiText PPL 6.36 and 65.96 average zero-shot accuracy versus 6.46 PPL and 59.96 for SliceGPT.
- ModHiFi-U achieves 0.2% forget accuracy on CIFAR-10 ResNet-50 without fine-tuning in 10 seconds, with 92.98% remain accuracy, versus 3.54% forget accuracy and 363 seconds for Jia et al.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ModHiFi-P instead of DFPC at the reported CIFAR-10 pruning setting | ResNet-50 structured pruning · CIFAR-10 | accuracy: 90.25 -> 91.02 | Helped |
| Use ModHiFi-P instead of L2 w/ ST at the reported CIFAR-10 pruning setting | ResNet-50 structured pruning · CIFAR-10 | accuracy: 90.12 -> 91.02 | Helped |
| Use ModHiFi-P instead of DFPC (30) | ResNet-50 structured pruning · ImageNet | accuracy: 75.9 -> 76.70 | Helped |
| Use ModHiFi-P instead of DFPC (54) | ResNet-50 structured pruning · ImageNet | CPU Speedup: 2.37x -> 2.42x | Helped |
| Use Alpaca rather than WikiText-2 for calibration (ablation) | ModHiFi-P on Llama-2-7B at 10% sparsity · Llama-2-7B, 10% sparsity | Average task accuracy: 64.23 -> 65.96 | Helped |
| Use Alpaca rather than WikiText-2 for calibration (ablation) | ModHiFi-P on Llama-2-7B at 20% sparsity · Llama-2-7B, 20% sparsity | Average task accuracy: 57.16 -> 60.62 | Helped |
| Use Alpaca rather than WikiText-2 for calibration (ablation) | ModHiFi-P on Llama-2-7B at 30% sparsity · Llama-2-7B, 30% sparsity | Average task accuracy: 48.63 -> 52.44 | Helped |
| Add Gaussian noise (standard deviation 0.01) to 20% of HiFi components rather than perturbing 80% of non-HiFi components (ablation) | ResNet-50 · CIFAR-10 | accuracy drop: around 12% vs 1% | Hurt |
| Add Gaussian noise (standard deviation 0.02) to 50% of HiFi components rather than 50% of non-HiFi components (ablation) | ResNet-50 · CIFAR-10 | accuracy drop: 85% vs around 1.4% | Hurt |
| Use higher-quality synthetic calibration samples (ablation) | ModHiFi-P · Structured pruning experiments | sparsity-accuracy tradeoff | Helped |
| Fine-tune ModHiFi-U for three epochs on synthetic samples (ablation) | Swin-Transformer classwise unlearning · CIFAR-10 | unlearning performance | Helped |
| Fine-tune ModHiFi-U for 10 epochs using synthetic samples rather than no fine-tuning (ablation) | Swin-Transformer classwise unlearning · CIFAR-10 | forget accuracy | Helped |
Surprising findings
- Well-trained transformers are empirically found to be Lipschitz continuous on bounded domains, contrary to prior assertions in the literature.
- On Swin-T, ModHiFi-U without fine-tuning falls well short of the state of the art (8.83% forget accuracy, 73.57% remain accuracy).
Limitations
- Theoretical bounds rely on local Lipschitz continuity, which is not guaranteed at initialization and only empirically verified for trained models.
- Optimality of singleton selection holds only under uncorrelated input contributions, an assumption that rarely holds exactly in practice.
- Attention-specific analysis is left for future work; the analysis focuses on the FFN; Swin-T unlearning without fine-tuning underperforms baselines.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet WikiText-2 Alpaca
Benchmarks: ImageNet CIFAR-10 CIFAR-100 WikiText-2 perplexity ARC-e ARC-c PIQA WinoGrande HellaSwag
Metrics: top-1 accuracy FLOP reduction parameter reduction CPU speedup GPU speedup perplexity zero-shot accuracy forget accuracy retain accuracy unlearning time
Baselines: GReg-2 OTO DepGRAPH ThiNet DFPC L2 norm pruning SliceGPT ShortGPT Gradient Ascent Jia et al. unlearning
Builds on: Optimal Brain Surgeon-style reconstruction pruning ShortGPT SliceGPT DFPC
Closest papers by meaning
- Patch Rebirth: Fast and Transferable Model Inversion of Vision Transformers (NeurIPS 2026 · similarity 0.87)
- Learning Where to Edit Vision Transformers (NeurIPS 2024 · similarity 0.86)
- DisCEdit: Model Editing by Identifying Discriminative Components (NeurIPS 2024 · similarity 0.86)
- Elastic ViTs from Pretrained Models without Retraining (NeurIPS 2025 · similarity 0.86)
- Split Adaptation for Pre-trained Vision Transformers (CVPR 2025 · similarity 0.85)
- Sparse Model Inversion: Efficient Inversion of Vision Transformers for Data-Free Applications (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.