Research map / Architectures and efficiency / Quantization
I-BERT: Integer-only BERT Quantization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use INT8 integer-only quantization and fine-tune the model | RoBERTa-Base · GLUE dev average | average score: +0.3 | Helped |
| Use INT8 integer-only quantization and fine-tune the model | RoBERTa-Large · GLUE dev average | average score: +0.5 | Helped |
| Use integer-only INT8 inference rather than FP32 inference | BERT-Base on Tesla T4 · Latency evaluation | speedup: 3.08 | Helped |
| Use integer-only INT8 inference rather than FP32 inference | BERT-Large on Tesla T4 · Latency evaluation | speedup: 3.56 | Helped |
| Replace GELU with h-GELU (ablation) | RoBERTa-Large · QNLI, SST-2, MRPC, and RTE | accuracy: 94.4 -> 94.3; 96.3 -> 96.0; 92.6 -> 92.8; 85.9 -> 84.8 | Mixed |
| Replace GELU with i-GELU (ablation) | RoBERTa-Large · QNLI, SST-2, MRPC, and RTE | accuracy: 94.4 -> 94.5; 96.3 -> 96.4; 92.6 -> 93.0; 85.9 -> 87.0 | Helped |
| Replace h-GELU with i-GELU (ablation) | RoBERTa-Large · QNLI, SST-2, MRPC, and RTE | accuracy: 94.3 -> 94.5; 96.0 -> 96.4; 92.8 -> 93.0; 84.8 -> 87.0 | Helped |
| Approximate GELU with i-GELU instead of h-GELU | GELU function · [-4, 4] approximation range | L² distance: 0.031 -> 0.0082 | Helped |
| Approximate GELU with i-GELU instead of h-GELU | GELU function · [-4, 4] approximation range | L∞ distance: 0.068 -> 0.018 | Helped |
Closest papers by meaning
- MA-BERT: Towards Matrix Arithmetic-only BERT Inference by Eliminating Complex Non-Linear Functions (ICLR 2023 · similarity 0.91)
- Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT (AAAI 2020 · similarity 0.90)
- ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (NeurIPS 2022 · similarity 0.89)
- PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination (ICML 2020 · similarity 0.89)
- Norm Tweaking: High-Performance Low-Bit Quantization of Large Language Models (AAAI 2024 · similarity 0.85)
- SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.