Research map / Architectures and efficiency / Quantization
QT-DoG: Quantization-Aware Training for Domain Generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QT-DoG, which uses quantization-aware training (LSQ, 7-bit) as an implicit regularizer that finds flatter minima for domain generalization, and EoQ, an ensemble of quantized models that beats state-of-the-art DG ensembles at lower memory.
Problem
Domain generalization suffers from overfitting to source domains; flatter minima are believed to help, but existing DG methods that find them (weight averaging, ensembles) incur high memory and compute cost.
Approach
Treat weight quantization noise (quantization-aware training with LSQ at 7 bits after 2000 steps) as a structured uniform weight perturbation that acts as an implicit regularizer toward flatter minima, and ensemble several quantized models so the ensemble stays small in memory.
Key results
- EoQ (5 quantized ResNet-50 models, 1.1x size) reaches 68.4% average DomainBed accuracy vs 68.0% for EoA and DiWA (60 models).
- Single QT-DoG (7-bit, ResNet-50, 0.22x size) reaches 66.2% average vs 66.9% for SWAD on DomainBed; on PACS 87.8 vs 84.7 for ERM.
- On PACS, 7-bit quantization gives the best out-of-domain accuracy; 2-bit drops to 82.8%, below ERM's 84.7%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add 7-bit quantization-aware training to ERM | ResNet-50 · PACS | OOD accuracy: 84.7 ± 0.5 -> 87.8± 0.3 | Helped |
| ensemble independently trained quantized models | QT-DoG · DomainBed average | accuracy: 66.2 -> 68.4 | Helped |
| replace LSQ QAT with post-training OBC quantization (ablation) | QT-DoG · PACS | OOD accuracy: 87.8 ± 0.3 -> 83.7 ± 0.4 | Hurt |
| replace LSQ with INQ QAT (ablation) | QT-DoG · PACS | OOD accuracy: 87.8 ± 0.3 -> 87.4 ± 0.3 | Hurt |
| add QT-DoG to CORAL | CORAL · PACS | accuracy: 85.5 ± 0.6 -> 86.9 ± 0.2 | Helped |
| add QT-DoG to MixStyle | MixStyle · TerraInc | accuracy: 44.0 ± 0.4 -> 47.7 ± 0.2 | Helped |
| quantize ERM_ViT with QT-DoG | DeiT-Small ERM_ViT · PACS | accuracy: 84.3 ± 0.2 -> 86.2 ± 0.3 | Helped |
| use channelwise rather than layerwise quantization scale (ablation) | QT-DoG · PACS | OOD accuracy: 86.3 ± 0.4 -> 87.8 ± 0.3 | Helped |
| apply 7-bit quantization after 2000 steps rather than after 1000, 3000, or 4000 steps (ablation) | QT-DoG · PACS | OOD accuracy: 1000: 86.2 ± 0.4; 2000: 87.8 ± 0.3; 3000: 86.9 ± 0.4; 4000: 85.1 ± 0.3 | Helped |
| vary quantization bit precision from 7 bits to 8, 6, or 5 bits (ablation) | QT-DoG · PACS | OOD accuracy: 7: 87.8 ± 0.3; 8: 85.0 ± 0.1; 6: 86.5 ± 0.1; 5: 85.3 ± 0.4 | Mixed |
| vary quantization bit precision from 7 bits to 4, 3, or 2 bits (ablation) | QT-DoG · PACS | accuracy: 7: 87.8 ± 0.3; 4: 84.3 ± 0.3; 3: 83.3 ± 0.4; 2: 82.8 ± 0.2 | Hurt |
| add uniform weight noise instead of quantization noise (ablation) | ERM · PACS | OOD accuracy: Uniform(-0.0001, 0.0001): 82.9 ± 0.6; Uniform(-0.00005, 0.00005): 83.8 ± 0.5; Uniform(-0.00001, 0.00001): 85.1 ± 0.4; Un | Mixed |
Surprising findings
- Post-training quantization (OBC) reduces out-of-domain accuracy on PACS (83.7 vs 84.7 for no quantization), while QAT methods improve it.
- Uniform random weight noise with larger ranges hurts OOD accuracy (e.g. Uniform(-0.0001,0.0001) gives 82.9), while very small noise helps slightly.
Limitations
- Optimal bit-width (7 bits) was found on DomainBed and may not transfer to other datasets; 4-bit and lower did not yield satisfactory results.
- EoQ requires training multiple models, increasing training compute.
- Only uniform quantization was studied; mixed-precision was left for future work.
Experimental setup
Datasets: PACS VLCS OfficeHome TerraIncognita DomainNet WILDS Amazon Camelyon CIFAR-based none
Benchmarks: PACS VLCS OfficeHome TerraIncognita DomainNet Amazon Camelyon CLIP-based DG datasets
Metrics: out-of-domain accuracy in-domain accuracy local flatness average accuracy 10th percentile accuracy
Baselines: ERM SAM SWA SWAD SMA DiWA EoA CORAL MixStyle CLIPood INQ OBC
Builds on: DomainBed LSQ SWAD DiWA EoA
Closest papers by meaning
- Seeking Consistent Flat Minima for Better Domain Generalization via Refining Loss Landscapes (CVPR 2025 · similarity 0.90)
- Merge-Friendly Post-Training Quantization for Multi-Target Domain Adaptation (ICML 2025 · similarity 0.89)
- QuEST: Stable Training of LLMs with 1-Bit Weights and Activations (ICML 2025 · similarity 0.87)
- Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization (NeurIPS 2025 · similarity 0.87)
- GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance (ICML 2025 · similarity 0.87)
- DOGE: Domain Reweighting with Generalization Estimation (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.