Research map / Architectures and efficiency / Pruning and efficient tuning

Data-free Neural Representation Compression with Riemannian Neural Dynamics

ICML 2024 · Architectures and efficiency › Pruning and efficient tuning · approach: Low-rank compression and merging

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RieM, a Riemannian-metric parameterization that models neuron interactions as dynamical states, enabling data-free compression of pretrained vision models (ResNet, ViT/Swin, DETR) that beats prior data-free quantization and pruning methods.

Problem

Compressing pretrained neural networks without access to real data (no fine-tuning), while preserving accuracy; existing neuron-based dynamical views use Euclidean metrics that limit expressiveness.

Approach

Interpret each weight matrix entry as a Riemannian metric between learned low-dimensional neuron states (via displacement vectors and nonlinear summation), so the weight matrix is reconstructed from neuron embeddings and a small metric network, then fit data-free to the original weights; add shared correlation counts for concatenation and dyMerge for faster inference.

Key results

• On ResNet-50 at 12.28 MB (4/4-bit), RieM reaches 73.26% top-1 vs 72.09% for UDFC and 70.80% for SQuant, exceeding the original-data-free comparisons.
• On DETR with ResNet-50 at 43.6 MB (8-bit), RieM-DETR is -0.4 AP vs T-DETR -0.6 AP, while outperforming T-DETR at 33.4 MB 4-bit in most AP metrics.
• On Swin-T at 14.5 MB (4/8-bit), RieM achieves 76.30% top-1 vs 76.28% for PSAQ-ViT V2 at the same size, and 80.85% at 8/8-bit.

Measured effects reported in the paper

Surprising findings

• Data-free RieM at 4/8-bit on Swin-T slightly outperforms the data-driven Standard V2 quantization at the same size (76.30 vs 75.51).
• Symmetry and bilinearity of the metric are not necessary for good compression performance, per the ablation.

Experimental setup

Datasets: ImageNet-1k COCO MNIST CIFAR-100

Benchmarks: ImageNet-1k COCO object detection MNIST CIFAR-100

Metrics: Top-1 accuracy bbox AP model size (MB) FLOPs BOPs

Baselines: DFQ UDFC DDAQ DSG SQuant GDFQ OMSE Neuron Merge PSAQ-ViT V2 T-DETR Quant-DETR SVD-DETR

Builds on: Pei & Wang 2023 neuronal dynamics UDFC Neuron Merging PSAQ-ViT

Closest papers by meaning

• A Riemannian Framework for Learning Reduced-order Lagrangian Dynamics (ICLR 2025 · similarity 0.87)
• Geometry-Aware Similarity Metrics for Neural Representations on Riemannian and Statistical Manifolds (NeurIPS 2026 · similarity 0.86)
• Building Transformation Layers for Riemannian Neural Networks (NeurIPS 2026 · similarity 0.85)
• Motif-Aware Riemannian Graph Neural Network with Generative-Contrastive Learning (AAAI 2024 · similarity 0.85)
• Task-Induced Riemannian Metrics for Vision Transformer Feature Spaces (NeurIPS 2026 · similarity 0.85)
• Exploring the Enigma of Neural Dynamics Through A Scattering-Transform Mixer Landscape for Riemannian Manifold (ICML 2024 · similarity 0.85)