Research map / Architectures and efficiency / Quantization
MetaMix: Meta-State Precision Searcher for Mixed-Precision Activation Quantization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MetaMix, a NAS-style mixed-precision activation quantization method that alternates meta-learned (bit-meta) weight training with bit-search on fixed meta-state weights to reduce activation instability, then fine-tunes the selected configuration.
Problem
Mixed-precision activation quantization of efficient, hard-to-quantize networks (MobileNet-v2/v3, ResNet-18) suffers from activation instability when bit-widths change during joint bit selection and weight training, degrading bit-search quality and accuracy.
Approach
Alternate a bit-meta training step (weights trained by averaging losses over single-bit-width networks, MAML-inspired) with a bit-search step that learns per-layer architectural parameters on the fixed full-precision meta-state weights, then fine-tune weights and step sizes with the selected per-layer bit-widths.
Key results
- On MobileNet-v2 at 4-bit, MetaMix reaches 72.60% top-1 vs. 71.56% for PROFIT and 72.1% for N2UQ (single-precision comparison).
- On ResNet-18, MetaMix achieves 71.94% at 32.86 GBOPs vs. 70.6% at 35.17 GBOPs for DNAS, with 13.4 GPU hours vs. 40 GPU hours for DNAS.
- On MobileNet-v3, MetaMix reaches 74.24% top-1 vs. 73.81% for PROFIT at 4-bit precision.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| bit-meta training (meta-state weights) before bit search (ablation) | MetaMix activation distributions on MobileNet-v2 · MobileNet-v2 activation distribution | activation consistency across bit-widths | Helped |
| fix full-precision meta-state weights during bit-search training (vs. training weights) (ablation) | MetaMix bit selection on MobileNet-v2 · ImageNet-1K MobileNet-v2 ~5.3 GBOPs | Top-1 (%): +0.46 (72.39 vs 71.93) | Helped |
| fix meta-state during bit-search training (ablation) | MetaMix on MobileNet-v3 (large) · ImageNet-1K MobileNet-v3 3.86 GBOPs | Top-1 (%): fixed gives 74.03; unfixed does not converge (NC) | Helped |
| iterative bit-meta and bit-search training in 2nd epoch (vs. non-iterative) (ablation) | MetaMix on MobileNet-v3 (large) · ImageNet-1K MobileNet-v3 3.86 GBOPs | Top-1 (%): +0.12 (74.03 vs 73.91) | Helped |
| apply PROFIT's early stopping of depth-wise layers to MetaMix (ablation) | MetaMix on MobileNet-v2 · ImageNet-1K MobileNet-v2 | Top-1 (%) | No clear effect |
| lower weight bit-width from 4 to 3 and 2 bits (activation mixed 8/4/2 candidates) (ablation) | MetaMix on ResNet-18 · ImageNet-1K ResNet-18 | Top-1 (%): 71.94 (4-bit) -> 70.69 (3-bit) -> 69.45 (2-bit) | Hurt |
| MetaMix bit-meta/bit-search vs DNAS-style search (MetaMix proposed) | ResNet-18 bit search · ImageNet-1K ResNet-18 search cost | GPU hours: 13.4 vs 40 GPU hours (DNAS) | Helped |
| MetaMix (mixed-precision activation, 4-bit weights) vs single-precision PROFIT on MobileNet-v2 | MobileNet-v2 4-bit · ImageNet-1K MobileNet-v2 | Top-1 (%): 72.60 vs 71.56 | Helped |
| MetaMix vs SDQ without strong regularization on MobileNet-v2 | mixed-precision method comparison · ImageNet-1K MobileNet-v2 ~5.15 GBOPs | Top-1 (%): 72.14 vs 71.38 | Helped |
| MetaMix mixed-precision activation vs N2UQ with 1st/last layers quantized on ResNet-18 | 4-bit ResNet-18 · ImageNet-1K ResNet-18 | Top-1 (%): 71.94 vs 70.60 | Helped |
| MetaMix mixed-precision activation vs PROFIT with 8-bit quantized first-layer input on MobileNet-v3 | 4-bit MobileNet-v3 (large) · ImageNet-1K MobileNet-v3 (large) | Top-1 (%): 74.24 vs 71.57 | Helped |
| apply PROFIT-style fine-tuning to N2UQ-style quantized first/last layers (N2UQ† with 1st/last 8-bit activations) | N2UQ on ResNet-18 · ImageNet-1K ResNet-18 | Top-1 (%): 72.9 (unquantized) -> 70.27 (quantized 1st/last) | Hurt |
Surprising findings
- Fine-tuning-based stabilization of batch norm statistics (PROFIT's approach) did not help when applied on top of MetaMix, because sensitive depth-wise layers are assigned high precision.
- Under power-of-two-bit restriction, MetaMix outperforms original (unrestricted) results of DQ, DJPQ and SDQ on ResNet-18.
Limitations
- Activation-only mixed precision with a single weight bit-width in main results; weight mixed-precision only explored in supplementary.
- Power-of-two restriction on MobileNets uses 3-bit instead of 2-bit because 2-bit gives inferior accuracy.
- Stabilization via PROFIT-style fine-tuning did not yield gains on MetaMix.
Experimental setup
Datasets: ImageNet-1K
Benchmarks: ImageNet-1K
Metrics: top-1 accuracy BOPs GPU hours
Baselines: PACT DSQ LLSQ LSQ LSQ+BR OOQ LCQ SAT APoT PROFIT N2UQ DUQ
Builds on: LSQ DNAS MAML PROFIT NIPQ
Closest papers by meaning
- FracBits: Mixed Precision Quantization via Fractional Bit-Widths (AAAI 2021 · similarity 0.88)
- AMPA: Adaptive Mixed Precision Allocation for Low-Bit Integer Training (ICML 2024 · similarity 0.88)
- Learning from Loss Landscape: Generalizable Mixed-Precision Quantization via Adaptive Sharpness-Aware Gradient Aligning (ICML 2025 · similarity 0.87)
- Efficient Multi-bit Quantization Network Training via Weight Bias Correction and Bit-wise Coreset Sampling (NeurIPS 2025 · similarity 0.87)
- ScaleBITS: Scalable Bitwidth Search for Hardware-Aligned Mixed-Precision LLMs (NeurIPS 2026 · similarity 0.86)
- MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.