Research map / Architectures and efficiency / Quantization
On Exact Bit-level Reversible Transformers Without Changing Architecture
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BDIA-transformer, an exact bit-level reversible transformer that keeps the standard architecture at inference by randomly sampling a gamma in {0.5,-0.5} per block during training, combined with activation quantization and stored side bits.
Problem
Existing reversible neural networks either need non-standard architectures or substantially modified standard ones to enable reversibility, and transformer training is memory-bound due to stored activations.
Approach
Treat each transformer block as an Euler integration step and import bidirectional integration approximation (BDIA) from diffusion inversion; randomly sign-flip the averaging coefficient gamma in {0.5,-0.5} during training (acting as an ensemble of ODE solvers and regularizer), quantize activations to 2^-l precision, store 1 bit of side information per activation per block for exact reversibility,
Key results
- BDIA-ViT reaches 89.10% validation accuracy on CIFAR10 vs 88.15% for ViT and 86.22% for RevViT, with 693.4MB peak memory vs 1570.6MB for ViT.
- On CIFAR100, BDIA-ViT reaches 66.09% vs 61.86% for ViT and 61.89% for RevViT.
- BDIA-GPT2 trained on a 0.05% OpenWebText subset has significantly lower final validation loss than GPT2, which overfits.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with randomly sampled gamma values ±0.5 per block and sample | ViT · CIFAR10 validation | accuracy: 88.15$\pm$0.55 -> 89.10$\pm$0.38 | Helped |
| Train with randomly sampled gamma values ±0.5 per block and sample | ViT · CIFAR100 validation | accuracy: 61.86$\pm$0.47 -> 66.09$\pm$0.80 | Helped |
| Use online back-propagation with BDIA and activation quantization | BDIA-ViT · CIFAR10 | peak memory: 1570.6MB -> 693.4MB | Helped |
| Use online back-propagation with BDIA and activation quantization | BDIA-ViT · CIFAR100 | peak memory: 1570.6MB -> 693.4MB | Helped |
| Replace ViT with RevViT | ViT · CIFAR10 validation | accuracy: 88.15$\pm$0.55 -> 86.22$\pm$0.42 | Hurt |
| Replace ViT with RevViT | ViT · CIFAR100 validation | accuracy: 61.86$\pm$0.47 -> 61.89$\pm$0.31 | No clear effect |
| Train with gamma values ±0.5 instead of gamma 0.0 (ablation) | BDIA-ViT · CIFAR10 validation | accuracy: 88.15$\pm$0.55 -> \textbf{89.12}$\pm$0.22 | Helped |
| Train with gamma values ±0.6 instead of gamma 0.0 (ablation) | BDIA-ViT · CIFAR10 validation | accuracy: 88.15$\pm$0.55 -> 88.89$\pm$0.15 | Helped |
| Train with gamma values ±0.25 instead of gamma 0.0 (ablation) | BDIA-ViT · CIFAR10 validation | accuracy | Helped |
| Train with nonzero gamma values rather than gamma 0.0 | BDIA-transformer · English-French translation validation | validation loss | Helped |
| Train with BDIA gamma random variables | GPT2 · OpenWebText validation, 0.05% training subset | validation loss | Helped |
Surprising findings
- Training loss of BDIA-ViT is higher than baselines across all epochs, yet validation accuracy is higher at the end, indicating regularization.
- RevViT often yields inferior or comparable validation performance to standard ViT.
Limitations
- Requires storing 1 bit of side information per activation per block, and so uses more memory than RevViT.
- Error accumulation makes direct reversal with gamma=±0.5 unstable without quantization; gamma=±0.25 needs 2 bits and gamma=±0.6 is hard to make exactly reversible.
- Larger gamma magnitude slows training speed.
Experimental setup
Datasets: CIFAR10 CIFAR100 English-French translation corpus OpenWebText (0.05% subset)
Benchmarks: CIFAR10 CIFAR100 English-French translation OpenWebText validation loss
Metrics: validation accuracy peak training memory validation loss
Baselines: ViT RevViT GPT2 conventional transformer
Builds on: BDIA diffusion inversion RevViT Dropout nanoGPT
Closest papers by meaning
- KroQuant: Kronecker-Structured Block Transforms for Efficient Post-Training Quantization of Diffusion Transformers (NeurIPS 2026 · similarity 0.85)
- PTQ4DiT: Post-training Quantization for Diffusion Transformers (NeurIPS 2024 · similarity 0.85)
- The Quantization Benefits of Residual-Free Transformers (NeurIPS 2026 · similarity 0.85)
- DiRotQ: Rotation-Aware Quantization for 4-bit Diffusion Transformers (NeurIPS 2026 · similarity 0.85)
- Q-DiT: Accurate Post-Training Quantization for Diffusion Transformers (CVPR 2025 · similarity 0.84)
- Predictor-Corrector Enhanced Transformers with Exponential Moving Average Coefficient Learning (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.