Research map / Learning theory and algorithms / Prediction and decision losses
Stochastic Forward-Forward Learning through Representational Dimensionality Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a negative-sample-free Forward-Forward goodness function based on the effective dimensionality of noisy neural responses (minimize per-sample ED over dropout copies, maximize ED across the batch), with energy-style mean-squared-output inference.
Problem
Forward-Forward learning relies on carefully designed negative samples and a goodness function (sum of squared activations) that ignores correlated variability between neurons; the paper targets a biologically plausible, local, negative-sample-free alternative to backpropagation.
Approach
Defines goodness as the effective dimensionality (uncentered second-moment participation ratio) of neural responses, with a consistency term minimizing ED over dropout-generated noisy copies of each input and a diversity term maximizing ED of the batch-averaged responses; inference uses the mean of squared classifier outputs over noisy variants.
Key results
- Proposed method reaches 99.31 ± 0.07% on MNIST, 76.96 ± 0.73% on CIFAR-10 and 53.29 ± 1.02% on CIFAR-100 (5 runs), above Original FF, CaFo FF, and DFA on the reported datasets but below BP (99.33, 82.50, 61.28).
- Setting alpha to 0 (only diversity term) lowers performance, especially on CIFAR-10 and CIFAR-100; alpha above 0.5 makes first-layer kernels collapse to similar patterns.
- Dropout noise strength and sample size matter: moderate dropout (p 0.1-0.3) with larger sample sizes gives best results in the appendix tables.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the dimensionality-compression objective instead of the original FF goodness function | Forward-Forward learning · MNIST, CIFAR-10, and CIFAR-100 | accuracy | Helped |
| Use the dimensionality-compression objective rather than BP under the reported setup | three-layer CNN · MNIST | accuracy: 99.33 ± 0.04 vs 99.31 ± 0.07 | No clear effect |
| Use the dimensionality-compression objective rather than BP under the reported setup | three-layer CNN · CIFAR-10 and CIFAR-100 | accuracy: 82.50 ± 0.09 vs 76.96. ± 0.73; 61.28 ± 0.25 vs 53.29 ± 1.02 | Hurt |
| Use the dimensionality-compression objective rather than EBL | three-layer CNN · MNIST, CIFAR-10, and CIFAR-100 | accuracy: 99.56 vs 99.31 ± 0.07; 89.6 vs 76.96. ± 0.73; 65.8 vs 53.29 ± 1.02 | Hurt |
| Set α=0.0, relying only on ED_d and omitting the ED_c consistency term (ablation) | dimensionality-compression method · CIFAR-10 and CIFAR-100 | accuracy | Hurt |
| Set α above 0.5 (ablation) | first convolutional block · CIFAR-10 | cosine orthogonality score | Hurt |
| Use mean output or a regular feedforward pass instead of mean squared output for inference (ablation) | dimensionality-compression classifier · classification tasks in the inference ablation | accuracy | Hurt |
| Add noisy sampling to supervised goodness optimization (ablation) | Sup goodness optimization · classification tasks in the supervised ablation | accuracy | Helped |
| Replace unsupervised noisy-copy training with label-supervised goodness optimization (ablation) | dimensionality-compression method · classification tasks in the supervised ablation | accuracy | Hurt |
| Use Fixed or Random projection dimensionality instead of Graded projection (ablation) | dimensionality-compression method · classification tasks in the projection ablation | accuracy | No clear effect |
| Omit the projection operation (ablation) | dimensionality-compression method · classification tasks in the projection ablation | accuracy | Hurt |
| Vary dropout probability and noisy-copy sample size (ablation) | dimensionality-compression method · MNIST, CIFAR-10, and CIFAR-100 | accuracy | Mixed |
Surprising findings
- Using the mean of squared outputs over noisy variants as prediction score performs slightly better than averaging outputs, suggesting variance carries label information.
- Adding more than three convolutional blocks degraded performance, which the authors attribute to collapse of class-specific features.
Limitations
- Does not achieve state-of-the-art accuracy; gap to BP widens on CIFAR-10 and CIFAR-100.
- Adding more than three convolutional blocks degrades performance; scaling to deeper networks or complex datasets not demonstrated.
- Noise sampling (N=20 copies) increases computational cost.
Experimental setup
Datasets: MNIST CIFAR-10 CIFAR-100
Benchmarks: MNIST CIFAR-10 CIFAR-100
Metrics: validation accuracy
Baselines: Original FF CaFo FF CwC FF DFA Soft Hebbian Hard Hebbian GIM EBL
Builds on: Forward-Forward energy-based learning Greedy InfoMax Cascaded Forward Hebbian winner-take-all
Closest papers by meaning
- Forward Learning with Top-Down Feedback: Empirical and Analytical Characterization (ICLR 2024 · similarity 0.86)
- Inferring stochastic low-rank recurrent neural networks from neural data (NeurIPS 2024 · similarity 0.86)
- The Price of Locality: Why Forward-Forward Underperforms Backpropagation? (NeurIPS 2026 · similarity 0.85)
- Critical feature learning in deep neural networks (ICML 2024 · similarity 0.84)
- Diversify, Contextualize, and Adapt: Efficient Entropy Modeling for Neural Image Codec (NeurIPS 2024 · similarity 0.84)
- Selectivity and Shape in the Design of Forward-Forward Goodness Functions (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.