Research map / Architectures and efficiency / RNNs and normalization
Training Verification-Friendly Neural Networks via Neuron Behavior Consistency
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes neuron behavior consistency (NBC), a training regularizer that keeps neuron activation states consistent within an input neighborhood, yielding networks with fewer unstable neurons that are faster to formally verify while keeping accuracy.
Problem
Formal verification of neural networks via branch-and-bound is slow because many unstable neurons cause exponential branching; existing verification-friendly training methods either post-process weights (costly, limited), depend on shifting bounds, or use heavy IBP certified training.
Approach
Add a regularizer that maximizes, via PGD-style inner adversarial search, the consistency of neuron pre-activation signs (approximated by cosine similarity per hidden layer plus a KL term on outputs) between an input and its perturbed neighbor, so that fewer neurons are unstable and bounds tighten, without needing IBP or bound computations.
Key results
- On MNIST M3 at epsilon=0.2, the proposed method verifies 46.3% of properties UNSAT versus 0.2%-2.3% for the other methods, with stable neuron ratio 68.7%.
- Combining NBC with Madry on MNIST M3 at epsilon=0.2 raises UNSAT% by 83.2 points and cuts verification time by 94.4 s.
- On CIFAR-10 C3 at 2/255, the method reaches 25.6% UNSAT and 80.1% stable neurons versus 2.5% and 21.9% for ReLU Stable, at a small accuracy cost (77.0% vs 78.0%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add neuron behavior consistency training | MNIST Conv Big (M3) · MNIST ε=0.2 | UNSAT%: 2.3 -> 46.3 | Helped |
| Add neuron behavior consistency training | MNIST Conv Big (M3) · MNIST ε=0.2 | Stable%: 4.3 -> 68.7 | Helped |
| Add neuron behavior consistency training | Fashion-MNIST Conv Big (M3) · Fashion-MNIST ε=0.3 | UNSAT%: 0.0 -> 17.4 | Helped |
| Add neuron behavior consistency training | CIFAR-10 Conv Big (C3) · CIFAR-10 ε=8/255 | Stable%: 0.2 -> 43.3 | Helped |
| Combine NBC with Madry training | MNIST Conv Big (M3) · MNIST ε=0.2 | UNSAT%: +83.2 | Helped |
| Combine NBC with ReLU Stable training | MNIST Conv Big (M3) · MNIST ε=0.2 | Time (s): -102.5 seconds | Helped |
| Combine NBC with existing training methods | Networks across MNIST, Fashion-MNIST, and CIFAR-10 · Combined-method experiments | Test accuracy | Hurt |
| Apply NBC training | CIFAR-10 networks · CIFAR-10 ε=2/255, close-accuracy comparison | Stable%: up to 80.1% | Helped |
Surprising findings
- Combining NBC with existing methods such as Madry or ReLU Stable yields large verification gains on the largest model, but at a cost in accuracy and PGD robustness.
- Applying stricter constraints to layers may reduce accuracy while increasing the proportion of stable neurons, so constraints are mainly placed on layers with fewer neurons.
Limitations
- Accuracy and PGD robustness are slightly lower than baselines, especially when combined with existing methods.
- Verification of the largest models (M3, C3) still times out for many properties at larger radii.
- Hyperparameters (beta, gamma scaling) are tuned per model on a validation set, and the method's underlying mathematical justification is only heuristically argued.
Experimental setup
Datasets: MNIST Fashion-MNIST CIFAR-10
Benchmarks: MNIST Fashion-MNIST CIFAR-10
Metrics: UNSAT% Stable% verification time Time_U+T PGD-100 accuracy test accuracy
Baselines: TRADES Madry ReLU Stable
Builds on: TRADES Madry adversarial training ReLU Stable loss alpha,beta-CROWN
Closest papers by meaning
- VNN: Verification-Friendly Neural Networks with Hard Robustness Guarantees (ICML 2024 · similarity 0.89)
- Certified but Private: Scalable Zero-Knowledge Proofs for the Formal Verification of Neural Networks (NeurIPS 2026 · similarity 0.88)
- SDP-CROWN: Efficient Bound Propagation for Neural Network Verification with Tightness of Semidefinite Programming (ICML 2025 · similarity 0.87)
- Verification Learning: Make Unsupervised Neuro-Symbolic System Feasible (ICML 2025 · similarity 0.86)
- Verifying Neural Networks with Reinforcement Learning (NeurIPS 2026 · similarity 0.86)
- NN4SysBench: Characterizing Neural Network Verification for Computer Systems (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.