Research map / Architectures and efficiency / RNNs and normalization
Selectivity and Shape in the Design of Forward-Forward Goodness Functions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically explores goodness-function design for Forward-Forward training and proposes selective (top-k, entmax-weighted energy) and shape-sensitive (burstiness/excess kurtosis, higher moments) goodness functions, combined with FFCL label injection, reaching 89.0% on Fashion-MNIST and 98.2% on MNIST.
Problem
Forward-Forward learning relies on a local goodness function, but sum-of-squares has been the only one studied; the principles for what makes a good goodness function are unexplored.
Approach
Goodness should depend on the shape of activations rather than total energy: top-k and entmax-weighted energy measure peak activity, while scale-invariant excess kurtosis and higher central moments reward heavy-tailed activation distributions.
Key results
- FFCL + moment-p=6 reaches 89.04% on Fashion-MNIST (4x2000), +32.6pp over the SoS baseline (56.41%).
- FFCL + burstiness reaches 98.18±0.08% on MNIST over 5 seeds versus 89.17±0.30% for SoS.
- Burstiness with standard FF gives 88.11% on Fashion-MNIST, outperforming FFCL + entmax-1.5 (87.12%).
Surprising findings
- Smooth activations (GELU, Swish) degrade SoS performance relative to ReLU, attributed to dense background activity inflating the energy.
- A 2x500 network with top-k goodness (76.65%) outperforms the 4x2000 SoS baseline (56.41%) on Fashion-MNIST.
Limitations
- Combinatorial ablation uses a single seed; multi-seed validation only for a subset of configurations.
- Absolute accuracy remains below backpropagation on Fashion-MNIST and no convolutional or larger-image architectures are tested.
- Hyperparameters are shared across methods without tuning per goodness function; entmax is roughly 7x slower.
Experimental setup
Datasets: MNIST Fashion-MNIST CIFAR-10 USPS SVHN EMNIST-Letters
Benchmarks: MNIST Fashion-MNIST CIFAR-10 USPS SVHN EMNIST-Letters
Metrics: test accuracy
Baselines: sum-of-squares goodness softmax-energy-margin game-theoretic goodness backpropagation
Builds on: Forward-Forward algorithm FFCL alpha-entmax independent component analysis
Closest papers by meaning
- Stochastic Forward-Forward Learning through Representational Dimensionality Compression (NeurIPS 2025 · similarity 0.84)
- Asymptotics of Learning with Deep Structured (Random) Features (ICML 2024 · similarity 0.84)
- Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimization (ICLR 2024 · similarity 0.84)
- Shortcut Features as Top Eigenfunctions of NTK: A Linear Neural Network Case and More (NeurIPS 2025 · similarity 0.84)
- Feature Learning Dynamics in Infinite-Depth Neural Networks (NeurIPS 2026 · similarity 0.83)
- From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learning (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.