Research map / Optimization / Optimizers
Synaptic Weight Distributions Depend on the Geometry of Plasticity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows with mirror descent theory that synaptic weight distributions reflect the synaptic geometry (potential), so log-normal weight data argue against Euclidean plasticity and suggest a way to test geometry experimentally.
Problem
Computational neuroscience models of synaptic plasticity implicitly assume Euclidean (gradient descent) geometry of synaptic changes, which is unjustified; it is unclear how to determine the brain's synaptic geometry from data.
Approach
Using mirror descent, the final weight distribution in the dual space (given by the potential's gradient) equals initial dual weights plus a Gaussian term that is independent of loss and data, so the potential that makes weight changes Gaussian identifies the synaptic geometry; applied to log-normal spine data this rules out Euclidean geometry.
Key results
- On linear regression with negative-entropy potential, ΔCDF stays at or below 0.2 and converges toward zero as width grows, confirming Gaussian dual-space weight changes.
- When finetuning pretrained ImageNet networks (ShuffleNet v2, EfficientNet, ResNet, CORnet) under three potentials, weight changes were near-Gaussian (ΔCDF < 0.2), and negative entropy gave the best fit only for negative-entropy-trained networks.
- Experimental spine-volume data are consistent with a negative-entropy geometry under a two-point initialization, and with a 3-norm geometry under a mixture initialization, which allows ruling out Euclidean geometry.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the training potential to transform weight changes into its dual space | Mirror descent in linear regression · Linear regression | ΔCDF from the standard normal: ≤ 0.2 | Helped |
| Increase the number of weights | Mirror descent in linear regression · Linear regression | ΔCDF from the standard normal | Helped |
| Evaluate changes using a potential close to the training potential | Mirror descent with negative entropy or p-norm potentials · Linear regression potential-change experiment | Gaussianity of dual-space weight changes | Helped |
| Evaluate negative-entropy-trained changes using a 3-norm potential | Mirror descent with negative entropy · Linear regression potential-change experiment | Gaussianity of dual-space weight changes | Hurt |
| Fine-tune using 2-norm, 3-norm, or negative-entropy geometry | Pretrained ShuffleNet v2, EfficientNet, ResNet, and CORnet · ImageNet validation subsets | ΔCDF from the standard normal: <0.2 | Helped |
| Use a hypothesized potential close to the training potential | Fine-tuned pretrained deep networks · ImageNet validation subsets | Gaussianity of dual-space weight changes | Helped |
| Use negative entropy as the hypothesized potential | Negative-entropy-trained networks · ImageNet validation subsets | Gaussian fit of dual-space weight changes | Helped |
| Use negative entropy as the hypothesized potential | Networks trained with potentials other than negative entropy · ImageNet validation subsets | Gaussian fit of dual-space weight changes | Hurt |
| Use 2-norm rather than the training 3-norm potential | 3-norm-trained EfficientNets 0, 1, 3 and CORnet-S · ImageNet validation subsets | Gaussian fit of dual-space weight changes | Mixed |
| Use Euclidean geometry rather than a non-Euclidean geometry | Synaptic weight distributions · Experimental spine-volume distribution | Fit to observed log-space distribution | Hurt |
Surprising findings
- The 2-norm (Euclidean) solution was slightly more Gaussian than the 3-norm solution for some 3-norm-trained EfficientNets and CORnet-S, though the differences were small.
- The experimental spine data can be fit by multiple potentials depending on the assumed initialization, so geometry cannot be identified without knowing initial weights.
Limitations
- Theory requires linearized/lazy regime and small, weakly correlated inputs; deep network results are empirical extensions.
- Identifying geometry from experimental data requires knowing initial weights; otherwise multiple potentials fit the data.
- Finetuning violates several theorem assumptions (non-iid initial weights, cross-entropy without unique finite root).
Experimental setup
Datasets: ImageNet validation subsets synthetic linear regression data published spine volume distribution (electron microscopy, rodent)
Benchmarks: ImageNet
Metrics: integral of absolute CDF difference (ΔCDF) from standard nor relative magnitude of dual-space weight change (Δφ)
Baselines: gradient descent (Euclidean 2-norm)
Builds on: mirror descent exponentiated gradient neural tangent kernel natural gradient descent
Closest papers by meaning
- A theory of weight distribution-constrained learning (NeurIPS 2022 · similarity 0.88)
- Model Based Inference of Synaptic Plasticity Rules (NeurIPS 2024 · similarity 0.87)
- Learning the Plasticity: Plasticity-Driven Learning Framework in Spiking Neural Networks (NeurIPS 2025 · similarity 0.85)
- How connectivity structure shapes rich and lazy learning in neural circuits (ICLR 2024 · similarity 0.84)
- Improving Deep Learning Speed and Performance Through Synaptic Neural Balance (AAAI 2025 · similarity 0.83)
- Comparing noisy neural population dynamics using optimal transport distances (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.