Research map / Architectures and efficiency / Attention and transformers
BiPFT: Binary Pre-trained Foundation Transformer with Low-Rank Estimation of Binarization Residual Polynomials
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BiPFT, the first binary pretrained foundation transformer for NLU, pretrained with 1-bit weights and activations plus a low-rank data-driven estimator of self-attention binarization residual polynomials, and finetuned without task-specific distillation.
Problem
Task-specific binary BERTs rely on full-precision teacher distillation and heavy hyperparameter search, are unstable, and lose learning ability without distillation; there is no binary foundation model for NLU.
Approach
Pretrain a binary BERT-style transformer (BiPFT-A) with MLM, NSP and task-agnostic distillation so finetuning needs no FP teacher; then analyze binarization error in self-attention as residual polynomials and model them with rank-1 low-rank estimators (BiPFT-B) learned during pretraining.
Key results
- BiPFT-B reaches 70.8 average GLUE dev score without task-specific distillation, versus 55.4 for the binary baseline (+15.4 points) and 68.1 for the distilled baseline*.
- BiPFT-A without distillation (69.2 avg) surpasses the distilled, hyperparameter-searched baseline* (68.1) by 1.1 points.
- Binary BiPFT-B saves about 56x operations and 28x memory versus FP BERT-base at 1-bit activations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| binary pretraining rather than training the binary baseline from scratch | Binary BERT · GLUE dev | average score: 55.4 -> 69.2 | Helped |
| add low-rank residual polynomial estimators to BiPFT-A | BiPFT-A · GLUE dev | average score: 69.2 -> 70.8 | Helped |
| add attention-value estimator only (ablation) | BiPFT-A · GLUE dev | average score: 69.2 -> 69.9 | Helped |
| add key-query estimator only (ablation) | BiPFT-A · GLUE dev | average score: 69.2 -> 70.2 | Helped |
| combine key-query and attention-value estimators (ablation) | BiPFT-A · GLUE dev | average score: 69.2 -> 70.8 | Helped |
| add estimators without binary pretraining (ablation) | Baseline · GLUE dev | average score: 55.4 -> 54.9 | Hurt |
| increase estimator rank from 1 to 2 (ablation) | BiPFT-B · GLUE dev | average score: 70.8 -> 69.7 | Hurt |
| increase estimator rank from 1 to 4 (ablation) | BiPFT-B · GLUE dev | average score: 70.8 -> 69.2 | Hurt |
| add LoRA alone (ablation) | BiPFT-A · GLUE dev | average score: 69.2 -> 69.9 | Helped |
| combine LoRA with residual polynomial estimators (ablation) | BiPFT-B · GLUE dev | average score: 70.8 -> 68.0 | Hurt |
| use full-precision pretrained initialization (ablation) | BiPFT-B · GLUE dev | average score: 69.6 -> 70.8 | Helped |
| use full-precision task-specific initialization (ablation) | Baseline* · GLUE dev | average score: 61.6 -> 68.1 | Helped |
Surprising findings
- Removing task-specific distillation from the binary baseline drops the average GLUE score by 13.9 points, and without pretraining the binary model collapses to near-random STS-B correlation.
- Increasing the rank of the residual estimators from 1 to 2 or 4 does not improve GLUE performance, suggesting overfitting.
Limitations
- Evaluated only on NLU/GLUE with base-sized BERT (110M); generation tasks left for future work.
- Data-driven estimators are data-hungry and do not help without pretraining.
- Baseline comparisons rely on the authors' re-run of prior methods under common settings.
Experimental setup
Datasets: BooksCorpus English Wikipedia GLUE
Benchmarks: GLUE
Metrics: accuracy F1 Pearson/Spearman correlation average GLUE score FLOPs model size
Baselines: BiT BiBERT BinaryBERT TernaryBERT Q-BERT Q2BERT baseline*
Builds on: BiT BiBERT BinaryBERT BERT LoRA
Closest papers by meaning
- PB-LLM: Partially Binarized Large Language Models (ICLR 2024 · similarity 0.85)
- Training Binary Neural Networks via Gaussian Variational Inference and Low-Rank Semidefinite Programming (NeurIPS 2024 · similarity 0.85)
- ARB-LLM: Alternating Refined Binarizations for Large Language Models (ICLR 2025 · similarity 0.84)
- How Do Nonlinear Transformers Learn and Generalize in In-Context Learning? (ICML 2024 · similarity 0.84)
- Transformers Learn to Implement Multi-step Gradient Descent with Chain of Thought (ICLR 2025 · similarity 0.84)
- BiPer: Binary Neural Networks using a Periodic Function (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.