Research map / Architectures and efficiency / Quantization
G-Net: A Provably Easy Construction of High-Accuracy Random Binary Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes G-Nets, floating-point networks trained on real data whose random sign (Gaussian or Rademacher) embeddings yield EHD G-Nets that run in binary Hamming space without binary training, with non-asymptotic consistency guarantees.
Problem
Hyperdimensional computing (HDC) models with binary embeddings lag far behind neural networks in accuracy, and training high-accuracy binary networks directly is hard; the paper asks how to build binary high-dimensional models whose accuracy provably approaches that of a real-valued network.
Approach
Use Grothendieck's identity so that a random sign embedding of weights and inputs gives inner products approximating an arc-sine nonlinearity; train a normalized-weight real-valued G-Net, then convert each layer to a binary EHD G-Net by sign-embedding weights and inputs with tall random matrices, with concentration-of-measure bounds on the discrepancy.
Key results
- EHD G-Net achieves over 99% accuracy on MNIST, rivaling real-valued CNNs, outperforming prior HDC models.
- Around 30% higher accuracy than prior HDC models on CIFAR-10 (HDC accuracies exceed 81% on CIFAR-10 per the paper).
- TASU-based EHD G-Net reaches 98.4% on MNIST while converging more slowly to its G-Net baseline than RASU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Gaussian embedding with Rademacher embedding in EHD G-Net | EHD G-Net · MNIST, CIFAR-10, WSS | accuracy | No clear effect |
| Use TASU activations (binary layer outputs) instead of RASU (ablation) | EHD G-Net · MNIST | accuracy | Hurt |
| Increase hyperdimension N of EHD G-Net | EHD G-Net · MNIST, CIFAR-10, WSS | accuracy | Helped |
| Use EHD G-Net (embedding of G-Net) instead of prior HDC models | HDC classifiers · CIFAR-10 | accuracy: almost 30% higher accuracy vs prior HDC models (abstract) | Helped |
Surprising findings
- Rademacher embeddings perform comparably to Gaussian embeddings despite weaker theoretical guarantees.
Limitations
- EHD G-Nets are high-dimensional, which may be impractical for some use cases.
- Cascade analysis is given only for base ASU activations; RASU and TASU cascade analyses are deferred.
- TASU-based EHD G-Nets converge more slowly to their G-Net baselines.
Experimental setup
Datasets: MNIST CIFAR-10 HAR-WSS
Benchmarks: MNIST CIFAR-10 HAR-WSS
Metrics: test accuracy
Baselines: classic HDC HoloGN Laplace HDC OnlineHD RFF-HDC convolutional neural network
Builds on: hyperdimensional computing random projection encoding Grothendieck's identity OnlineHD NeuralHD
Closest papers by meaning
- Binary Graph Neural Networks (CVPR 2021 · similarity 0.86)
- How to Train a Compact Binary Neural Network with High Accuracy? (AAAI 2017 · similarity 0.86)
- The High-Dimensional Geometry of Binary Neural Networks (ICLR 2018 · similarity 0.86)
- Fast and Slow Gradient Approximation for Binary Neural Network Optimization (AAAI 2025 · similarity 0.85)
- Training Binary Neural Networks via Gaussian Variational Inference and Low-Rank Semidefinite Programming (NeurIPS 2024 · similarity 0.85)
- HADAMRNN: BINARY AND SPARSE TERNARY ORTHOGONAL RNNS (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.