Research map / Architectures and efficiency / RNNs and normalization
Mind the Gap: Removing the Discretization Gap in Differentiable Logic Gate Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Gumbel Logic Gate Networks, which inject Gumbel noise into DLGN gate selection (with a straight-through estimator) to speed up training, reduce the discretization gap, and eliminate unused neurons, with a Hessian-trace smoothing argument.
Problem
Differentiable logic gate networks (DLGNs) train slowly and suffer a large discretization gap between soft (relaxed) training and discrete inference, with many neurons left unused.
Approach
Sample Gumbel-perturbed gate logits and select a hard gate in the forward pass, using a straight-through estimator with the Gumbel-Softmax relaxation in the backward pass; a lemma shows Gumbel noise implicitly regularizes the Hessian trace, smoothing the loss landscape.
Key results
- Gumbel LGNs converge 4.75x faster in iterations and 4.5x faster in wall-clock time than DLGNs on CIFAR-10.
- Discretization gap reduced by 98% and unused gates reduced from 49.81% to 0.00%.
- Runtime per iteration increases by about 4.6% on average with Gumbel noise.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Gumbel noise for gate selection | DLGN · CIFAR-10 | discretization gap: 98% | Helped |
| Use Gumbel noise for gate selection | DLGN · CIFAR-10, depth 12, width 256k | wall-clock training speed: 4.5× | Helped |
| Use Gumbel noise for gate selection | DLGN · CIFAR-10 | unused gates: 49.81% -> 0.00% | Helped |
| Add a straight-through estimator to the Gumbel gate selection (ablation) | Gumbel LGN · CIFAR-10, depth 12, width 256k | discretization gap | Helped |
| Add a straight-through estimator to the Gumbel gate selection (ablation) | Gumbel LGN · CIFAR-10, depth 12, width 256k | convergence speed | Hurt |
| Set temperature τ from 0.25 to 1.00 (ablation) | Gumbel LGN · CIFAR-10, depth 12, width 256k | final accuracy: 0.568 -> 0.575 | Helped |
| Set temperature τ from 0.25 to 1.00 (ablation) | Gumbel LGN · CIFAR-10, depth 12, width 256k | iterations to within 1% of maximum accuracy (×10³): 440 -> 918 | Hurt |
| Set temperature τ to 2.00 instead of 1.00 (ablation) | Gumbel LGN · CIFAR-10, depth 12, width 256k | final accuracy: 0.575 -> 0.480 | Hurt |
| Set temperature τ to 2.00 instead of 1.00 (ablation) | Gumbel LGN · CIFAR-10, depth 12, width 256k | iterations to within 1% of maximum accuracy (×10³): 918 -> 1342 | Hurt |
| Set temperature τ to 0.01 instead of 0.25 (ablation) | Gumbel LGN · CIFAR-10, depth 12, width 256k | final accuracy: 0.568 -> 0.546 | Hurt |
| Set temperature τ to 0.01 instead of 0.25 (ablation) | Gumbel LGN · CIFAR-10, depth 12, width 256k | iterations to within 1% of maximum accuracy (×10³): 440 -> 972 | Hurt |
Surprising findings
- Adding Gumbel noise alone (soft Gumbel without ST) affects convergence and discretization, and the ST estimator slows convergence while further shrinking the gap.
- Accuracy is non-monotone in temperature: tau values above 1 or below 0.1 converge much slower.
Limitations
- Evaluation limited mainly to CIFAR-10 and CIFAR-100; no error bars due to compute constraints (single seeds).
- Temperature tau requires tuning to balance convergence speed and accuracy.
- Theoretical analysis relies on simplifications and does not fully treat the discretization gap.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 MNIST EMNIST FashionMNIST KMNIST QMNIST
Benchmarks: CIFAR-10 CIFAR-100
Metrics: test accuracy discretization gap wall-clock training time unused gates (entropy-based) Hessian trace
Baselines: DLGN (softmax relaxation) Soft Gumbel (no straight-through)
Builds on: Differentiable Logic Gate Networks Gumbel-Softmax DARTS Sharpness-Aware Minimization
Closest papers by meaning
- Convolutional Differentiable Logic Gate Networks (NeurIPS 2024 · similarity 0.90)
- BOLD: Boolean Logic Deep Learning (NeurIPS 2024 · similarity 0.85)
- Differentiable Weightless Neural Networks (ICML 2024 · similarity 0.85)
- On the Hardness of Training Deep Neural Networks Discretely (AAAI 2025 · similarity 0.84)
- LogicMP: A Neuro-symbolic Approach for Encoding First-order Logic Constraints (ICLR 2024 · similarity 0.84)
- Towards training digitally-tied analog blocks via hybrid gradient computation (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.