Research map / Architectures and efficiency / Quantization
Enhancing Diversity for Data-free Quantization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EnhancingDFQ, a generator-based data-free post-training quantization method that uses multi-layer features, attention-based mixup, and normalization-flow attention plus regulation losses to produce diverse calibration data and adaptive clip ranges.
Problem
Generator-based data-free quantization suffers from mode collapse, yielding low-diversity synthetic calibration data, and fixed clip ranges that fit poorly to the long-tailed activation distributions of that data.
Approach
Builds label embeddings from multi-layer features of the full-precision model, mixes classes with attention-based mixup (multi-layer features mixer), uses normalization-flow-based attention for intra-class variation, and adds a cosine-similarity regulation loss on activations plus a learnable-clip-range distillation loss for the quantized model.
Key results
- On ImageNet 3w3a/4w4a CNN quantization, achieves ResNet-50 4w4a 71.20% top-1, outperforming TexQ (70.72%) and RIS (70.61%).
- On Transformers at 4w8a, reports DeiT-B 82.51% and Swin-S 82.51% top-1 (full precision 81.85 and 83.20 respectively per the table headers), with gains over CLAMP-ViT.
- Ablation on ResNet-50 4w4a shows removing all components drops accuracy from 71.20 to 52.12%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EnhancingDFQ instead of TexQ for 4w4a quantization | ResNet-50 · ImageNet | top-1 accuracy: 70.72 -> 71.20 | Helped |
| Use EnhancingDFQ instead of TexQ for 3w3a quantization | ResNet-20 · CIFAR-100 | top-1 accuracy: 55.87 -> 56.17 | Helped |
| Use EnhancingDFQ instead of HAST for 3w3a quantization | ResNet-18 · CIFAR-100 | top-1 accuracy: 51.15 -> 52.27 | Helped |
| Use EnhancingDFQ instead of TexQ for 4w4a quantization | ResNet-18 · CIFAR-100 | top-1 accuracy: 67.73 -> 68.13 | Helped |
| Use EnhancingDFQ instead of TexQ for 4w4a quantization | MobileNetV2 · ImageNet | top-1 accuracy: 67.07 -> 67.10 | Helped |
| Include the full set of proposed components rather than an ablated configuration (ablation) | EnhancingDFQ on ResNet-50 · ImageNet, 4w4a | top-1 accuracy: 69.14 -> 71.20 | Helped |
| Include the full set of proposed components rather than an ablated configuration (ablation) | EnhancingDFQ on DeiT-T · ImageNet, 4w8a | top-1 accuracy: 68.37 -> 69.89 | Helped |
| Include the full set of proposed components rather than an ablated configuration (ablation) | EnhancingDFQ on ResNet-50 · ImageNet, 4w4a | top-1 accuracy: 52.12 -> 71.20 | Helped |
| Include the full set of proposed components rather than an ablated configuration (ablation) | EnhancingDFQ on DeiT-T · ImageNet, 4w8a | top-1 accuracy: 64.83 -> 69.89 | Helped |
Surprising findings
- The quantized ViT-B at 8w8a (84.19 on ImageNet listing) slightly exceeds the full-precision ViT-B, which the authors attribute to attention-based mixup acting like Mixup training.
Limitations
- Evaluated only on image classification with CNN and Transformer backbones; applicability to large models and other tasks is untested.
- Gains are modest on some settings (e.g., DeiT-T and Swin-T where differences are not significant per paired t-test at 0.1).
Experimental setup
Datasets: ImageNet CIFAR-100
Benchmarks: ImageNet CIFAR-100
Metrics: top-1 accuracy GPU hours pairwise image similarity
Baselines: GDFQ ARC Qimera HAST IntraQ AdaSG AdaDFQ TexQ RIS Standard PSAQ-ViT Standard V2
Builds on: Qimera GDFQ Mixup normalization flow PSAQ-ViT
Closest papers by meaning
- Diversifying Sample Generation for Accurate Data-Free Quantization (CVPR 2021 · similarity 0.92)
- Quantization without Tears (CVPR 2025 · similarity 0.87)
- Retraining-Free Model Quantization via One-Shot Weight-Coupling Learning (CVPR 2024 · similarity 0.87)
- Adaptive Dataset Quantization (AAAI 2025 · similarity 0.87)
- Sharpness-Aware Data Generation for Zero-shot Quantization (ICML 2024 · similarity 0.86)
- Robustness-Guided Image Synthesis for Data-Free Quantization (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.