Research map / Architectures and efficiency / Pruning and efficient tuning
MCNC: Manifold-Constrained Reparameterization for Neural Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MCNC, a reparameterization that restricts network weight chunks to a k-dimensional nonlinear manifold (a frozen random sinusoidal MLP generator wrapping a hypersphere), achieving high compression for ViT, ResNet, and LLaMA-2 adapters.
Problem
Storing and transmitting large models is costly, and existing compression (pruning, quantization, low-rank PEFT) either trades accuracy at high compression rates or constrains parameters in ways that limit extreme compression.
Approach
Reparameterize each d-dimensional chunk of weights as beta times a unit vector produced by a frozen, randomly initialized sine-activated MLP generator mapping a k-dimensional input (k much smaller than d) onto the hypersphere, so only k+1 scalars per chunk are trained with standard gradient descent; generalizes PRANC and NOLA (random subspaces) to nonlinear manifolds.
Key results
- On ViT-Ti at 10% of model size, MCNC reaches 72.9% accuracy vs. 67.2% for PLATON pruning (baseline 83.5%).
- On ResNet-18 ImageNet-100 at 1% size, MCNC w/ LoRA reaches 71.7% vs. 64.7% for NOLA (baseline 82.1%).
- On LLaMA-2 7B (4-bit) Alpaca fine-tuning, MCNC matches NOLA on MMLU (45.9 vs 45.5) with 1.37 vs 2.56 GFLOPs for adapter reconstruction and about 2x throughput.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MCNC instead of PLATON at 10% model size | ViT-Ti · ImageNet-100 | accuracy: 67.2 -> 72.9 | Helped |
| Use MCNC instead of PLATON at 5% model size | ViT-Ti · ImageNet-100 | accuracy: 55.0 -> 69.1 | Helped |
| Add LoRA to MCNC (ablation) | ResNet-18 · ImageNet-100, 10% model size | accuracy: 78.0±0.0 -> 80.3±0.5 | Helped |
| Add LoRA to MCNC (ablation) | ResNet-18 · ImageNet-100, 1% model size | accuracy: 63.4±0.2 -> 71.7±0.3 | Helped |
| Add LoRA to MCNC (ablation) | ResNet-20 and ResNet-56 · CIFAR-10 and CIFAR-100 | accuracy | Helped |
| Use MCNC instead of NOLA for adapter reconstruction | LLaMA-2 7B and 13B · Alpaca / MMLU | MMLU accuracy: 45.5 -> 45.9; 54.8 -> 55.0 | No clear effect |
| Use MCNC instead of NOLA | LLaMA-2 7B and 13B · Alpaca inference | throughput (samples/sec): 3.1 -> 6.2; 1.7 -> 3.9 | Helped |
| Use Sine activation rather than no activation (ablation) | MCNC · MNIST | accuracy: 81.6 ± 0.5 -> 84.6 ± 0.7 | Helped |
| Use ReLU rather than no activation (ablation) | MCNC · MNIST | accuracy: 81.6 ± 0.5 -> 76.0 ± 2.8 | Hurt |
| Increase generator input frequency from 1.0 to 4.0 (ablation) | MCNC · MNIST | accuracy: 81.9 ± 0.4 -> 85.1±0.2 | Helped |
| Train the generator rather than leave it random (ablation) | MCNC · CIFAR-10/100, ResNet-20/56 | accuracy: 82.3 ± 0.3 -> 82.4 ± 04; 78.7 ± 0.2 -> 79.2 ± 02; 34.7 ± 0.2 -> 36.1 ± 0.6; 35.2 ± 0.1 -> 36.7 ± 0.5 | Helped |
| Increase hidden size from 16 to 512 (ablation) | MCNC MLP · MNIST | accuracy: 81.1 ± 0.3 -> 85.2±0.2 | Helped |
Surprising findings
- Many common activations (ReLU, Leaky ReLU, ELU) perform worse than having no activation at all in the generator ablation, while Sine performs best.
- Training the generator only marginally improves accuracy over a random generator (e.g., 34.7 to 36.1 on CIFAR-100 R20).
Limitations
- The optimal generator architecture, loss, and regularizers were not fully explored.
- Training large language models from scratch with MCNC was not evaluated due to computational constraints.
- Pruning methods outperform MCNC at higher compression rates on ViT-S (e.g., at 25% size PLATON-style results are close, and pruning wins at larger budgets).
Experimental setup
Datasets: ImageNet-100 CIFAR-10 CIFAR-100 MNIST Alpaca
Benchmarks: ImageNet-100 CIFAR-10 CIFAR-100 MNIST MMLU
Metrics: top-1 accuracy compression rate (percentage of model size) parameter count validation loss throughput (samples/sec) GFLOPs for adapter reconstruction CPU-to-GPU transfer time
Baselines: Magnitude pruning PLATON PRANC NOLA STR DPF SuRP LoRA (rank=1)
Builds on: PRANC NOLA LoRA QLoRA SWGAN
Closest papers by meaning
- On Compressing Deep Models by Low Rank and Sparse Decomposition (CVPR 2017 · similarity 0.88)
- Compressing Large Language Models using Low Rank and Low Precision Decomposition (NeurIPS 2024 · similarity 0.86)
- Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models (NeurIPS 2024 · similarity 0.86)
- Effective Interplay between Sparsity and Quantization: From Theory to Practice (ICLR 2025 · similarity 0.86)
- Breaking the Compression Ceiling: Data-Free Pipeline for Ultra-Efficient Delta Compression (NeurIPS 2025 · similarity 0.86)
- Distribution-Aware Tensor Decomposition for Compression of Convolutional Neural Networks (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.