Research map / Learning theory and algorithms / Prediction and decision losses
Error Correction Output Codes for Robust Neural Networks against Weight-errors: A Neural Tangent Kernel Point of View
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses neural tangent kernel theory to explain why non-one-hot error correction output codes (ECOCs) improve DNN robustness to weight errors, derives a normalized-codeword-distance threshold guarantee, and proposes two ECOC construction methods (direct optimization and Hadamard-based selection).
Problem
One-hot output codes lose accuracy under weight errors (e.g., from NVCiM hardware defects), and prior ECOC work is empirical with no theory explaining when and why ECOCs improve DNN robustness or how to design them.
Approach
Analyzes DNNs in the NTK regime to show that ECOC replaces the l2 decoding metric with a Mahalanobis metric defined by the codeword correlation matrix, and bounds output perturbation under weight noise so that the normalized codeword distance (divided by sqrt of code length) determines error correction; the bound yields a decoding-correctness condition. Uses this to design codes trading off orthog
Key results
- On CIFAR100 with nL=1024 and VGG16, Method 2 reaches 61.71% at weight-error 0.008 versus 58.73% for random codes and 3.72% for one-hot code.
- On MNIST with nL=128, Method 1 reaches 58.35% at omega=0.05 vs 46.39% for repetition codes (nL=130 with one-hot at 58.43%), showing gains under weight errors with clean accuracy unchanged.
- The abstract claims proposed codes outperform existing ECOCs by up to 7%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Method 1 instead of Hadamard codes | ECOC classifier on CIFAR10 with AlexNet · CIFAR10, ω=0.04, nL=16 | accuracy: 17.43 ± 0.25 -> 19.58 ± 0.25 | Helped |
| Use Method 1 instead of Hadamard codes | ECOC classifier on MNIST with MLP · MNIST, ω=0.05, nL=16 | accuracy: 60.49 ± 0.86 -> 58.35 ± 0.94 | Hurt |
| Use Method 2 instead of Hadamard codes | ECOC classifier on CIFAR100 with VGG16 · CIFAR100, ω=0.013, nL=1024 | accuracy: 28.63 ± 1.21 -> 36.13 ± 1.41 | Helped |
| Use Method 2 instead of Hadamard codes | ECOC classifier on Tiny ImageNet with ResNet-50 · Tiny ImageNet, ω=0.008, nL=1024 | accuracy: 34.02 ± 0.64 -> 38.17 ± 0.43 | Helped |
| Use Method 2 instead of Method 1 (ablation) | ECOC classifier on CIFAR100 with VGG16 · CIFAR100, ω=0.013, nL=1024 | accuracy: 33.83 ± 0.76 -> 36.13 ± 1.41 | Helped |
| Use Method 2 instead of Method 1 (ablation) | ECOC classifier on CIFAR10 with AlexNet · CIFAR10, ω=0.04, nL=1024 | accuracy: 34.89 ± 0.88 -> 34.71 ± 0.75 | Hurt |
| Increase code length (ablation) | ECOC classifiers under weight-errors · MNIST and CIFAR10 | accuracy | Helped |
| Vary the number of complement pairs in Method 2 (ablation) | ECOC classifier on CIFAR100 with VGG16 · CIFAR100, nL=1024 | accuracy | Mixed |
Surprising findings
- Hadamard and random codes perform nearly identically at large code lengths, as correlation and distance statistics concentrate.
- Method 1 (direct optimization) beats Method 2 on small tasks but loses on large tasks, attributed to local minima.
Limitations
- Experiments are simulation only, without implementation on real hardware accelerators.
- Theory relies on NTK assumptions requiring infinite network width, which is strong for practical networks.
- Bound guarantees robustness to weight errors but not ground-truth correctness.
Experimental setup
Datasets: MNIST CIFAR10 CIFAR100 Tiny ImageNet
Benchmarks: MNIST CIFAR10 CIFAR100 Tiny ImageNet
Metrics: classification accuracy under Gaussian weight-error
Baselines: one-hot code repetition code random code Hadamard code
Builds on: neural tangent kernel error correcting output codes Hadamard codes
Closest papers by meaning
- COLA: Orchestrating Error Coding and Learning for Robust Neural Network Inference Against Hardware Defects (ICML 2023 · similarity 0.90)
- Scalable design of Error-Correcting Output Codes using Discrete Optimization with Graph Coloring (NeurIPS 2022 · similarity 0.87)
- Towards the Training of Deeper Predictive Coding Neural Networks (ICML 2026 · similarity 0.85)
- Benchmarking Predictive Coding Networks -- Made Simple (ICLR 2025 · similarity 0.84)
- Only Strict Saddles in the Energy Landscape of Predictive Coding Networks? (NeurIPS 2024 · similarity 0.83)
- Codebook Features: Sparse and Discrete Interpretability for Neural Networks (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.