Research map / Architectures and efficiency / Pruning and efficient tuning
Forget the Data and Fine-Tuning! Just Fold the Network to Compress
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes model folding, a data-free, fine-tuning-free compression method that clusters structurally similar neurons/channels with k-means and repairs variance statistics with data-free approximate REPAIR (Fold-AR) or Deep Inversion (Fold-DIR).
Problem
Compressing neural networks (CNNs and LLMs) without training data or fine-tuning, where existing structured pruning and merging methods either need data for statistics correction or degrade severely at high sparsity.
Approach
Cast channel/neuron merging within a single network as k-means clustering of weight rows (jointly over a layer and its successor), which minimizes the approximation error, and then restore variance-collapse with data-free statistic repair (Fold-AR analytic correlation estimate, or Fold-DIR using one Deep Inversion batch for BatchNorm updates).
Key results
- On ResNet18 with CIFAR10, model folding keeps over 80% accuracy at 70% sparsity while structured magnitude pruning barely exceeds random chance.
- On LLaMA-7B at 20% sparsity, model folding without data or fine-tuning reaches 50.17 average zero-shot accuracy versus 63.00 for Wanda_sp (calibration data) and 65.55 for the dense model.
- Fold-AR and Fold-DIR closely match data-driven REPAIR (Fold-R) and surpass IFM by a significant margin at high sparsity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| compress with Model Folding instead of FLAP | LLaMA-7B · 20% prune ratio, WikiText2 | perplexity: 6.87 -> 13.33 | Hurt |
| compress with Model Folding instead of Wanda_sp | LLaMA-7B · 20% prune ratio, zero-shot tasks | Average: 63.00 -> 50.17 | Hurt |
| fine-tune layer norms after folding (ablation) | LLaMA-7B · 20% prune ratio, WikiText2 | perplexity: 13.33 -> 8.95 | Helped |
| fine-tune layer norms after folding (ablation) | LLaMA-7B · 20% prune ratio, zero-shot tasks | Average: 50.17 -> 55.33 | Helped |
| compress with Model Folding instead of FLAP | LLaMA2-7B · 20% prune ratio, WikiText2 | perplexity: 6.87 -> 17.46 | Hurt |
| compress with Model Folding instead of FLAP | LLaMA2-7B · 20% prune ratio, zero-shot tasks | Average: 61.85 -> 50.03 | Hurt |
| use Wanda with out-of-distribution Hungarian-word calibration data instead of C4 | LLaMA-7B · WikiText2 | perplexity: 8.22 -> 13.98 | Hurt |
Surprising findings
- Model folding without fine-tuning on LLaMA-7B at 20% sparsity is well behind data-driven baselines on average accuracy (50.17 vs 63.00 for Wanda_sp), though perplexity is competitive for some methods.
- Out-of-distribution calibration data (random Hungarian or Ukrainian text) degrades Wanda's WikiText2 perplexity from 8.22 to about 13.9, showing data-driven methods are sensitive to calibration domain.
Limitations
- Effectiveness may be limited in networks with low redundancy.
- Does not optimize sparsity levels per layer; uniform sparsity is used.
- Without fine-tuning, folded LLaMA-7B lags data-driven pruning methods on zero-shot average accuracy.
Experimental setup
Datasets: CIFAR10 CIFAR100 ImageNet WikiText2 C4
Benchmarks: CIFAR10 CIFAR100 ImageNet WikiText2 BoolQ WinoGrande ARC-e ARC-c
Metrics: accuracy perplexity average zero-shot accuracy variance ratio
Baselines: IFM INN structured magnitude pruning ZipIt! LLM-Pruner FLAP Wanda_sp SliceGPT ShortGPT Magnitude Pruning
Builds on: REPAIR weight matching Deep Inversion IFM
Closest papers by meaning
- Cut Less, Fold More: Model Compression through the Lens of Projection Geometry (ICLR 2026 · similarity 0.91)
- Model Preserving Compression for Neural Networks (NeurIPS 2022 · similarity 0.88)
- In defense of parameter sharing for model-compression (ICLR 2024 · similarity 0.87)
- Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism (NeurIPS 2025 · similarity 0.86)
- Breaking the Compression Ceiling: Data-Free Pipeline for Ultra-Efficient Delta Compression (NeurIPS 2025 · similarity 0.86)
- ESPACE: Dimensionality Reduction of Activations for Model Compression (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.