Research map / Architectures and efficiency / RNNs and normalization
The Sign Code: The Hidden Binary Nature of Deep Networks
Niclas A Göring, Shuofeng Zhang, Roi Holtzman, yoonsoo nam, Ard Louis
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Investigates whether classification in overparameterized vision networks depends more on the binary sign code of neuron outputs or on their magnitudes.
Problem
Whether the decision of trained deep vision classifiers depends more on the sign pattern of neuron outputs (sign code) or on their magnitudes.
Approach
Defines the sign code as the set of all neuron signs in a trained network and argues that, for overparameterized vision classifiers with sign-preserving activations, classification is overwhelmingly determined by the sign code rather than magnitudes.
Key results
- Abstract claims that for overparameterized vision classifiers with sign-preserving activations, classification depends overwhelmingly on the sign code rather than magnitudes (no numbers stated in the provided abstract).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase network depth and width (ablation) | FFNN/ResNet/ViT, SiLU · CIFAR-10 | sign gap: negligible (< 0.03) once both large | Improved |
| flip 20% of neuron signs (ablation) | ResNet-18 and FFNN · CIFAR-10 / SVHN | accuracy: to ~10% (chance) | Hurt |
| replace every magnitude with random values while enforcing clean sign code (ablation) | ResNet-18 · CIFAR-10 | accuracy: 93-95% retained | No clear effect |
| replace every magnitude with random values while enforcing clean sign code (ablation) | FFNN with ReLU/GELU/SiLU · SVHN | accuracy: drop ~36 / ~34 / ~32 pp | Hurt |
| replace every magnitude with random values while enforcing clean sign code (ablation) | FFNN with tanh · SVHN | accuracy: drop ~2 pp | No clear effect |
| add within-class sign-code repulsion regularizer (ablation) | SiLU network · CIFAR-10 | test accuracy: train 97%, test falls to ~40% | Hurt |
| add within-class sign-code repulsion regularizer (ablation) | tanh network · classification | test accuracy | No clear effect |
| use only last-layer sign code with kNN instead of full model | overparameterized networks · MNIST/CIFAR-10/CIFAR-100, 540 models | test accuracy | No clear effect |
| use nearest class mean on features (neural collapse test) instead of full model | overparameterized networks · CIFAR-10 | test accuracy: NCM peaks at 60-70% vs ~80% full model | Hurt |
| use depth 1 network with width 4096 (ablation) | FFNN · CIFAR-10 | gate-domination score Z1: Z1 < 0.23 | No clear effect |
Surprising findings
- Classification is overwhelmingly determined by the binary sign code rather than neuron magnitudes, per the abstract.
Closest papers by meaning
- Sign Gradient Descent-based Neuronal Dynamics: ANN-to-SNN Conversion Beyond ReLU Network (ICML 2024 · similarity 0.84)
- Codebook Features: Sparse and Discrete Interpretability for Neural Networks (ICML 2024 · similarity 0.84)
- On the Optimization and Generalization of Two-layer Transformers with Sign Gradient Descent (ICLR 2025 · similarity 0.84)
- Self-Explainable Graph Transformer for Link Sign Prediction (AAAI 2025 · similarity 0.83)
- A Signed Graph Approach to Understanding and Mitigating Oversmoothing (NeurIPS 2025 · similarity 0.83)
- The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
