Research map / Architectures and efficiency / RNNs and normalization
Ex Uno Pluria: Insights on Ensembling in Low Precision Number Systems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LPE-BSR, which builds training-free ensembles by sampling Bernoulli stochastic-rounding variants of pre-trained weights in an INT-5 low precision system, and shows gains over RTN, Bayesian methods (SWAG, IVON), and fast ensembling on ViT, CLIP and LLaMA-3 models.
Problem
Scaling deep ensembles to large pre-trained models is memory-costly, and fine-tuned or pre-trained solutions from a single checkpoint lie in one basin, limiting diversity; post-training quantization ignores the many good nearby solutions in low precision systems.
Approach
Treat quantization error as a source of ensemble diversity: sample ensemble members from a single pre-trained model by Bernoulli stochastic rounding onto a low precision (INT-B) grid, requiring no training, so each member is stored cheaply in low precision.
Key results
- On CLIP-ViT-L/14, LPE-BSR (INT-5) reduces NLL from 0.948 (FP32 pre-trained) to 0.929 and ECE from 0.049 to 0.028 with no training.
- On LLaMa-3-8B zero-shot MMLU, LPE-BSR (INT-5) lowers NLL from 1.03 to 0.923 and ECE from 0.160 to 0.087, while error rises slightly from .361 to .364.
- In fine-tuned comparisons with SWAG and IVON, LPE-BSR with INT-5 members matches or beats BMA with FP32 members while using fewer bits per member.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use INT-4 stochastic-rounding ensemble instead of INT-4 rounding-to-nearest | ViT-L/16 · ImageNet validation subset | NLL: .647 -> .600 | Helped |
| Use INT-4 stochastic-rounding ensemble instead of INT-4 rounding-to-nearest | ViT-T/16 · ImageNet validation subset | NLL: 1.23 -> 1.30 | Hurt |
| Use INT-4 stochastic-rounding ensemble instead of INT-4 rounding-to-nearest | ViT-L/16 · ImageNet validation subset | ERR: .167 -> .160 | Helped |
| Use INT-5 LPE-BSR instead of MAP alone | CLIP-ViT-L/14 fine-tuned with SWAG · ImageNet | NLL: .488\PM{.002} -> \BF{.473}\PM{.002} | Helped |
| Use INT-5 LPE-BSR instead of Bayesian model averaging | CLIP-ViT-L/14 fine-tuned with SWAG · ImageNet | NLL: .477\PM{.002} -> \BF{.473}\PM{.002} | Helped |
| Use INT-5 LPE-BSR instead of Bayesian model averaging | CLIP-ViT-L/14 fine-tuned with IVON · ImageNet | NLL: .475\PM{.001} -> \BF{.472}\PM{.001} | Helped |
| Ensemble stochastic-rounded weights in INT-5 | CLIP-ViT-L/14 · ImageNet | NLL: .948 -> .929 | Helped |
| Ensemble stochastic-rounded weights in INT-5 | CLIP-ViT-G/14 · ImageNet | NLL: .942 -> .927 | Helped |
| Ensemble stochastic-rounded weights in INT-5 | LLaMa-3 · MMLU | NLL: 1.03 -> .923 | Helped |
| Use LPE-BSR INT-5 instead of Gaussian-noise ensembling | CLIP-ViT-L/14 · ImageNet | NLL: \UL{.934} -> \BF{\UL{.929}} | Helped |
| Use LPE-BSR INT-5 instead of Monte Carlo dropout | CLIP-ViT-L/14 · ImageNet | NLL: \UL{.938} -> \BF{\UL{.929}} | Helped |
| Increase LPE-BSR precision from INT-5 to INT-4 (ablation) | CLIP-ViT-L/14 fine-tuned with SWAG · ImageNet | NLL: \BF{.473}\PM{.002} -> .513\PM{.002} | Hurt |
Surprising findings
- Lower-precision (INT-4) stochastic rounding increases ensemble diversity and can improve NLL for large ViT models even though individual members are less accurate.
- Gaussian and MCD training-free baselines give only marginal gains, while LPE-BSR outperforms them at lower memory cost.
Limitations
- Only the simplest symmetric uniform INT quantization was studied; other number systems were not explored.
- Fake quantization was used, so latency and real speedups were not measured.
- Gains on fine-tuned models are small in some metrics, e.g., ERR for SWAG and IVON is essentially unchanged.
Experimental setup
Datasets: ImageNet MMLU
Benchmarks: ImageNet MMLU
Metrics: NLL classification error ECE ensemble ambiguity memory budget
Baselines: RTN SWAG IVON SSE CSGLD Deep Ensembles Batch Ensembles Gaussian noise ensemble Monte Carlo Dropout
Builds on: Bayesian model averaging SWAG IVON snapshot ensembling Monte Carlo dropout
Closest papers by meaning
- Snapshot Ensembles: Train 1, Get M for Free (ICLR 2017 · similarity 0.87)
- PEP: Parameter Ensembling by Perturbation (NeurIPS 2020 · similarity 0.87)
- Deep Ensembles Work, But Are They Necessary? (NeurIPS 2022 · similarity 0.86)
- Rethinking LLM Ensembling from the Perspective of Mixture Models (ICML 2026 · similarity 0.86)
- Subsampled Ensemble Can Improve Generalization Tail Exponentially (NeurIPS 2025 · similarity 0.84)
- Theoretical Limitations of Ensembles in the Age of Overparameterization (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.