Research map / Probabilistic and scientific modeling / Variational and generative sampling
Training Binary Neural Networks via Gaussian Variational Inference and Low-Rank Semidefinite Programming
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VISPA, a Gaussian variational inference framework for BNN training that learns weight means plus a low-rank covariance factor (Burer-Monteiro SDP), recovering STE and weight clipping as a special case.
Problem
Training binarized neural networks is a combinatorial optimization problem; current STE-based latent-weight methods are heuristic and theoretically poorly understood, and ignore correlations between weights.
Approach
Relax BNN training to optimizing Gaussian distributions over weights with moment constraints (mean in [-1,1], diagonal of covariance fixed), solved via low-rank covariance factor Z learned by gradient descent with sign rounding; this explains STE and clipping theoretically and captures pairwise weight correlations.
Key results
- VISPA reaches 93.25% on CIFAR-10 VGG16 and 72.09% on CIFAR-100 VGG16 (1W32A), beating AdaSTE by 2.81 points on CIFAR-100.
- On ImageNet AlexNet 1W1A, VISPA reaches 51.1% top-1, 3.2 points above Quantization Networks.
- On ImageNet ResNet18 1W1A, VISPA reaches 62.1% top-1, 0.5 points above ReBNN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add deviation matrix Z to model weight correlations (ablation) | VISPA · CIFAR-10, VGG16, 1W32A | testing accuracy (%): 92.85 ± 0.09 -> 93.25 ± 0.11 | Helped |
| add deviation matrix Z to model weight correlations (ablation) | VISPA · CIFAR-10, ResNet18, 1W32A | testing accuracy (%): 95.03 ± 0.08 -> 95.05 ± 0.10 | No clear effect |
| add deviation matrix Z to model weight correlations (ablation) | VISPA · CIFAR-100, VGG16, 1W32A | testing accuracy (%): 70.12 ± 0.10 -> 72.09 ± 0.17 | Helped |
| add deviation matrix Z to model weight correlations (ablation) | VISPA · CIFAR-100, ResNet18, 1W32A | testing accuracy (%): 76.23 ± 0.36 -> 77.05 ± 0.41 | Helped |
| add deviation matrix Z to model weight correlations (ablation) | VISPA · Tiny-ImageNet, ResNet18, 1W32A | testing accuracy (%): 56.73 ± 0.33 -> 58.98 ± 0.28 | Helped |
| increase covariance rank K from 1 to 10 (ablation) | VISPA, ResNet18 · CIFAR-10, 1W1A | testing accuracy (%): 92.53 ± 0.11 -> 93.01 ± 0.07 | Helped |
| increase covariance rank K from 1 to 10 (ablation) | VISPA, VGG-Small · CIFAR-10, 1W1A | testing accuracy (%): 92.72 ± 0.09 -> 92.23 ± 0.09 | Hurt |
| increase covariance rank K from 1 to 10 (ablation) | VISPA, ResNet18 · CIFAR-100, 1W32A | testing accuracy (%): 76.45 ± 0.15; 77.05 ± 0.41 | Mixed |
| increase covariance rank K from 1 to 10 (ablation) | VISPA, VGG16 · CIFAR-100, 1W32A | testing accuracy (%): 71.09 ± 0.27 -> 72.26 ± 0.18 | Helped |
| use VISPA instead of the strongest baseline | AlexNet, ImageNet 1/1 · ImageNet | Top1 accuracy (%): 47.9% by 3.2% | Helped |
| use VISPA instead of the strongest baseline | ResNet18, ImageNet 1/1 · ImageNet | Top1 accuracy (%): 0.5% | Helped |
| use VISPA instead of DIR-Net | ResNet18, ImageNet 1/32 · ImageNet | Top5 accuracy (%): 87.9 -> 87.8 | Hurt |
Surprising findings
- Ablation shows the deviation matrix Z gives only modest gains on CIFAR-10 (e.g., 95.03 vs 95.05 on ResNet18), suggesting gains concentrate on harder datasets.
- Accuracy does not improve consistently with covariance rank K; VGG-Small on CIFAR-10 (1W1A) slightly declines as K grows.
Limitations
- null
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet
Metrics: top-1 accuracy top-5 accuracy
Baselines: BayesBiNN AdaSTE ReSTE DIR-Net ReCU BiPer ReBNN Bop Quantization Networks
Builds on: BayesBiNN straight-through estimator mean-field variational inference Burer-Monteiro low-rank SDP hyperplane rounding
Closest papers by meaning
- An Empirical study of Binary Neural Networks' Optimisation (ICLR 2019 · similarity 0.89)
- Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization (NeurIPS 2019 · similarity 0.89)
- Binarized Neural Networks (NeurIPS 2016 · similarity 0.88)
- Fast and Slow Gradient Approximation for Binary Neural Network Optimization (AAAI 2025 · similarity 0.87)
- A Study of Bayesian Neural Network Surrogates for Bayesian Optimization (ICLR 2024 · similarity 0.87)
- HADAMRNN: BINARY AND SPARSE TERNARY ORTHOGONAL RNNS (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.