Research map / Architectures and efficiency / Attention and transformers
COMCAT: Towards Efficient Compression and Customization of Attention-Based Vision Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Factorize combined head-level matrices (W^QK, W^VO) instead of individual Q/K/V/O matrices (ablation) | DeiT-small MHA layers (no fine-tuning) · ImageNet-1K | Top-1 accuracy: 79.81 vs 73.01 at 20% params; 63.56 vs 22.95 at 60% | Helped |
| Automatic rank selection (NAS-style) instead of fixed rank (ablation) | Low-rank DeiT-small compression · ImageNet-1K | Top-1 accuracy during training: qualitative: smaller accuracy loss, params converge to target | Helped |
| Use SVD instead of Tucker or Tensor Train for FFN low-rank compression (ablation) | DeiT-small FFN layers (no fine-tuning) · ImageNet-1K | Top-1 accuracy at 10% FFN params: 76.75 SVD vs 73.69 Tucker vs 0.81 Tensor Train | Helped |
| Low-rank MHA + rank search compared to pruning (UVC) for DeiT-small | DeiT-small compression · ImageNet-1K | Top-1 accuracy: +0.45% over UVC (79.27 vs 78.82 reported in text) | Helped |
| Low-rank MHA customization in cross-attention vs CustomDiffusion/DreamBooth | Stable Diffusion v1-4 personalization · Custom concepts (7 objects) training cost | Training time and extra storage: training time 193s vs 237s (CustomDiffusion) and 502s (DreamBooth); extra storage 6 MB vs 75 MB and 11565 MB | Helped |
| Low-rank MHA customization in cross-attention vs CustomDiffusion/DreamBooth | Stable Diffusion v1-4 personalization · Custom concepts FID (average over 7 concepts) | Average FID (lower is better): 101.23 vs 128.06 (CustomDiffusion) and 118.03 (DreamBooth) | Helped |
| Low-rank MHA customization on MS-COCO generation | Stable Diffusion v1-4 personalized models · MS-COCO FID | FID: 30.85-32.84 vs pre-trained 32.90 | No clear effect |
Closest papers by meaning
- Dense Vision Transformer Compression with Few Samples (CVPR 2024 · similarity 0.88)
- DeepCompress-ViT: Rethinking Model Compression to Enhance Efficiency of Vision Transformers at the Edge (CVPR 2025 · similarity 0.88)
- QT-ViT: Improving Linear Attention in ViT with Quadratic Taylor Expansion (NeurIPS 2024 · similarity 0.88)
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.87)
- Bi-ViT: Pushing the Limit of Vision Transformer Quantization (AAAI 2024 · similarity 0.87)
- MimiQ: Low-Bit Data-Free Quantization of Vision Transformers with Encouraging Inter-Head Attention Similarity (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.