Research map / Architectures and efficiency / RNNs and normalization
What should a neuron aim for? Designing local objective functions based on information theory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes trivariate infomorphic neurons whose local goal is a weighted sum of PID atoms (shared-exclusion redundancy) over feedforward, context and lateral inputs, and trains MNIST classifiers that approach backpropagation performance.
Problem
Local learning in neural networks is hard to interpret and usually underperforms global backpropagation; bivariate information-theoretic local objectives struggle on classification because neurons need a third signal (lateral) to self-organize.
Approach
Extends infomorphic neurons from two to three input classes (feedforward, context, lateral) and defines each neuron's local goal as a linear combination of 19 trivariate PID atoms (plus residual entropy), with goal weights set heuristically or optimized via TPE/CMA-ES, then trained by autograd through a differentiable shared-exclusion PID estimator.
Key results
- Trivariate infomorphic network (setup 2, sparse lateral connections) reaches a median test accuracy of 97.5% on MNIST with 2000 hidden neurons, versus 98.0% for backpropagation.
- For 100 hidden neurons, trivariate infomorphic networks outperform bivariate infomorphic networks with both heuristic and optimized goals on MNIST.
- Only four goal parameters are critically non-zero for high MNIST performance; the optimized goal weights the redundancy atom Π{F}{C} at 0.98 and penalizes Π{F}{L} at -0.99.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use sparse rather than all-to-all lateral connectivity (ablation) | Trivariate infomorphic network · MNIST, larger hidden layers | test accuracy: 97.5 %; 98.0 % | Helped |
| Replace the original activation function with linear activation (ablation) | Infomorphic network with 100 hidden neurons · MNIST | accuracy: 94.9\,\% and 95.1\,\% | No clear effect |
| Add a third input class for lateral connections to the hidden-layer neurons | Infomorphic network · MNIST, 100 hidden neurons | test accuracy | Helped |
| Replace the hidden-layer label context with feedback from the output layer (ablation) | Trivariate infomorphic network, setup 3 · MNIST | test accuracy | Mixed |
| Increase hidden-layer size in setup 1 beyond 500 neurons (ablation) | Trivariate infomorphic network, setup 1 · MNIST | test accuracy | Hurt |
| Use multiple hidden layers without further optimization of the goal or other hyperparameters (ablation) | Infomorphic network · MNIST test | accuracy: 97.2 \%; 97.1 \% | No clear effect |
| Add an output-feedback context to the multiple-hidden-layer architecture (ablation) | Infomorphic network, 1600-400-10 · MNIST test | accuracy: 97.7 \%; 97.5 % | Helped |
| Vary batch size (ablation) | Trivariate infomorphic network, 100 hidden neurons · MNIST | performance | No clear effect |
| Vary the number of discretization bins (ablation) | Trivariate infomorphic network · MNIST | training convergence | Mixed |
Surprising findings
- Replacing the label context with feedback from the output layer still gives solid performance, especially for large layers.
- Deeper infomorphic networks train out-of-the-box with the same goal parameters and match or slightly exceed single-hidden-layer accuracy in preliminary experiments.
Limitations
- Only a single fully connected hidden layer is evaluated in the main results; deeper networks are only preliminary.
- PID is computed on discretized (20-bin) variables, limiting information per neuron and requiring a plug-in estimator.
- Accuracy does not fully match backpropagation for large layers (97.5% vs 98.0%) and convergence degrades for dense lateral connections at large sizes.
Experimental setup
Datasets: MNIST CIFAR-10
Benchmarks: MNIST CIFAR-10
Metrics: test accuracy validation accuracy mean decrease in impurity
Baselines: backpropagation bivariate infomorphic neurons random hidden layer with step-function activation linear activation variant
Builds on: infomorphic neurons partial information decomposition shared-exclusion redundancy coherent infomax
Closest papers by meaning
- Shaping the distribution of neural responses with interneurons in a recurrent circuit model (NeurIPS 2024 · similarity 0.86)
- Nonlinear dynamics of localization in neural receptive fields (NeurIPS 2024 · similarity 0.85)
- Evaluating Neuron Explanations: A Unified Framework with Sanity Checks (ICML 2025 · similarity 0.84)
- Towards a "Universal Translator" for Neural Dynamics at Single-Cell, Single-Spike Resolution (NeurIPS 2024 · similarity 0.84)
- Range, not Independence, Drives Modularity in Biologically Inspired Representations (ICLR 2025 · similarity 0.84)
- Linear Explanations for Individual Neurons (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.