Research map / Architectures and efficiency / RNNs and normalization

DeepCompress-ViT: Rethinking Model Compression to Enhance Efficiency of Vision Transformers at the Edge

CVPR 2025 · Architectures and efficiency › RNNs and normalization · approach: Dataset condensation and flow estimation · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DeepCompress-ViT, which jointly trains an encoder-decoder weight codec with ViTs (Unified Compression Training) and uses a reordered matmul at test time to compress DeiT models ~14-17x for edge deployment.

Problem

ViTs are too large for on-chip memory on edge devices, and existing pruning and quantization methods lose substantial accuracy at the >10x compression ratios needed to fit the model on-chip.

Approach

Treat compression as learned encoding of weights with a lightweight linear decoder trained jointly with the ViT via MSE, CE and distillation losses, and at inference reorder the decoding matmul ((x·z)·wD) so the decoder adds no FLOPs when the compressed rank r satisfies a derived bound.

Key results

• DeepCompress-DeiT-S reaches 14.9x compression (84.1 MB to 5.64 MB) with 78.74% top-1 vs 79.72% for the uncompressed DeiT-S.
• Hardware simulation reports energy reductions of about 673x (DeiT-S) and 1475x (DeiT-B) with 8 MB on-chip memory, and latency reductions of 44.5x and 68.4x.
• On ImageNet, DeepCompress-ViT outperforms WDPruning (60.51%), LSQ (68.00%) and Q-ViT (71.90%) at comparable or higher compression.

Measured effects reported in the paper

Surprising findings

• The reported energy reductions (up to 1475x) are far larger than the compression ratio alone would suggest, attributed to reduced off-chip memory access in a simulated accelerator.

Limitations

• Hardware results come from a simulated accelerator (gem5-Aladdin, DRAMSim2) rather than a physical edge device.
• Accuracy drop on detection (APb 26.2 to 23.0) is non-trivial, and the compressed-object-detection comparison is only against DeiT-S.
• Encoder-decoder training requires 275 epochs with a distilled teacher, adding training cost.

Experimental setup

Datasets: ImageNet-1K COCO

Benchmarks: ImageNet-1K COCO

Metrics: top-1 accuracy model size (MB) compression ratio GFLOPs energy consumption (J) latency (s) APb APm

Baselines: WDPruning LSQ Q-ViT DeiT-S DeiT-B

Builds on: DeiT LSQ Q-ViT Mask R-CNN

Closest papers by meaning

• Unified Visual Transformer Compression (ICLR 2022 · similarity 0.90)
• Linearly Decomposing and Recomposing Vision Transformers for Diverse-Scale Models (NeurIPS 2024 · similarity 0.89)
• Dense Vision Transformer Compression with Few Samples (CVPR 2024 · similarity 0.88)
• GradQ-ViT: Robust and Efficient Gradient Quantization for Vision Transformers (AAAI 2025 · similarity 0.87)
• Bi-ViT: Pushing the Limit of Vision Transformer Quantization (AAAI 2024 · similarity 0.87)
• RPFQ-ViT: Rotated Phase-Frame Quantization for Extremely Low-Bit Weights in Vision Transformers (NeurIPS 2026 · similarity 0.87)