Research map / Architectures and efficiency / RNNs and normalization
Differentiable Weightless Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Differentiable Weightless Neural Networks (DWNs), multi-layer lookup-table networks trained by gradient descent via Extended Finite Difference, with learnable mapping, learnable reduction, and spectral regularization, evaluated on FPGA, microcontroller, ultra-low-cost chip, and tabular benchmarks.
Problem
Weightless neural networks (LUT-based, multiplication-free models) have been restricted to single layers because LUTs are discrete and not differentiable, and existing multi-layer methods (DiffLogicNet) scale double-exponentially with LUT size; they also use fixed random connections and popcount rea
Approach
Approximate gradients through n-input lookup tables with Extended Finite Difference, which weights all 2^n table positions by Hamming distance (cost O(2^n) rather than O(2^(2^n))), and add learnable input mappings, LUT-based learnable reduction replacing popcount, and a Fourier-based spectral regularizer for LUTs.
Key results
- On the Zynq Z-7045 FPGA, DWNs achieve geometric-average 20.7x lower latency, 12.3x higher throughput, and 121.6x lower energy per sample versus FINN, and 3.3x / 2.3x / 19.0x versus ULEEN.
- On the Elegoo Nano microcontroller, bit-packed DWNs are on average 5.4% more accurate than XGBoost; unpacked DWNs are 15% faster and 1.2% more accurate on average.
- On 11 tabular datasets, DWN achieves average rank 2.5 versus 3.4 for XGBoost and 7.5 for TabNet, with average L1 0.005.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Extended Finite Difference (ablation) | DWN with Finite Difference · MNIST, FashionMNIST, ToyADMOS/car; single-layer KWS and CIFAR-10 | accuracy: 96.15% -> 96.59%; 85.74% -> 86.88%; 87.73% -> 88.02%; 0.00%* gain on KWS and CIFAR-10 | Helped |
| add Learnable Mapping (ablation) | DWN with Finite Difference · MNIST, FashionMNIST, KWS, ToyADMOS/car, CIFAR-10 | accuracy: 2.15%; 2.20%; 17.91%; 0.79%; 6.99% | Helped |
| add Extended Finite Difference and Learnable Mapping (ablation) | DWN with Finite Difference · MNIST, FashionMNIST, KWS, ToyADMOS/car, CIFAR-10 | accuracy: 2.16%; 3.27%; 19.19%; 1.30%; 9.05% | Helped |
| add Spectral Normalization (ablation) | DWN · MNIST, FashionMNIST, KWS, ToyADMOS/car, skin-seg, higgs | accuracy: +0.06%; +1.06%; +2.43%; +0.64%; +0.18%; +3.91% | Helped |
| add Spectral Normalization (ablation) | DWN · phoneme | accuracy: -0.05% | Hurt |
| replace popcount with Learnable Reduction (ablation) | DWN for ultra-low-cost chips · phoneme, skin-seg, higgs, australian, nomao, segment, miniboone, christine, jasmine, sylvine, blood | NAND2 count: 257 -> 168; 346 -> 88; 231 -> 94; 36 -> 7; 210 -> 87; 160 -> 71; 121 -> 60; 97 -> 53; 114 -> 51; 159 -> 44; 156 -> 49 | Helped |
| use DWN instead of ULEEN | FPGA inference on FashionMNIST · FashionMNIST | test accuracy: 87.86 -> 89.12 | Helped |
| use DWN instead of FINN | FPGA inference on CIFAR-10 · CIFAR-10 | test accuracy: 80.10 -> 57.42 | Hurt |
| use DWN instead of XGBoost | Elegoo Nano throughput-optimized inference · Elegoo Nano datasets | accuracy: 1.2% more accurate on average; less accurate on higgs | Mixed |
| use bit-packed instead of unpacked DWN inference | DWN on Elegoo Nano · Elegoo Nano datasets | accuracy: 4.2% | Helped |
| use DWN instead of XGBoost | tabular data models · Tabular datasets | average rank: 3.4 -> 2.5 | Helped |
Surprising findings
- Parameter size is not a good predictor of hardware efficiency: an n=2 MNIST DWN with about 1/4 the parameter size of the n=6 model uses more than twice the area and energy on FPGA.
- Ablation on tabular/ultra-low-cost: DWN with learnable reduction uses far fewer NAND2 gates than popcount-based DiffLogicNet (e.g., 88 vs 610 on skin-seg).
Limitations
- Several n=2 DWN and DiffLogicNet models could not be synthesized on the FPGA due to routing congestion from irregular connectivity.
- Bit-packed microcontroller implementation is about 8.3x slower than XGBoost on average, and the unpacked version is less accurate on higgs.
- Evaluated only on tabular, small image, and audio tasks; not extended to CNN or Transformer architectures.
Experimental setup
Datasets: MNIST FashionMNIST KWS ToyADMOS/car CIFAR-10 JSC phoneme skin-seg higgs australian nomao segment
Benchmarks: MNIST FashionMNIST KWS ToyADMOS/car CIFAR-10 JSC tabular benchmark suite (11 datasets)
Metrics: test accuracy latency throughput energy per sample LUT count area NAND2 equivalent size average rank average L1 distance
Baselines: FINN ULEEN DiffLogicNet LogicNets PolyLUT NeuraLUT Tiny Classifier XGBoost CatBoost TabNet
Builds on: DiffLogicNet ULEEN Thermometer encoding XGBoost FINN
Closest papers by meaning
- Mind the Gap: Removing the Discretization Gap in Differentiable Logic Gate Networks (NeurIPS 2025 · similarity 0.85)
- Differentiable Learning of Generalized Structured Matrices for Efficient Deep Neural Networks (ICLR 2024 · similarity 0.83)
- Data-Efficient Learning with Neural Programs (NeurIPS 2024 · similarity 0.83)
- Differentiable Model Scaling using Differentiable Topk (ICML 2024 · similarity 0.83)
- The Effect of Weight Precision on the Neuron Count in Deep ReLU Networks (ICML 2024 · similarity 0.82)
- On the Hardness of Training Deep Neural Networks Discretely (AAAI 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.