Research map / Architectures and efficiency / Pruning and efficient tuning
FLOPS: Forward Learning with OPtimal Sampling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FLOPS, a plug-and-play Gaussian query allocator that adaptively distributes a fixed query budget across data points in a mini-batch for perturbation-based forward learning (LR and SPSA), with theoretical optimality results and experiments on ViT fine-tuning, black-box prompt tuning, and multimodal alignment.
Problem
Forward (perturbation-based) learning needs many queries per data point for low-variance gradient estimates, which hurts scalability; existing methods allocate queries equally across data points regardless of estimation difficulty.
Approach
Treat per-example query counts as a random allocation vector drawn from a Gaussian allocator whose parameters are optimized via the likelihood-ratio method to minimize summed gradient-estimate trace variance; reparameterize mean via loss (tanh-linear) and covariance via an RBF kernel on CLS-token cosine similarity, leaving only four allocator parameters.
Key results
- On ViT-base, OPS-LR reaches 65.8% on ImageNet versus 57.3% for equally allocated LR and 80.5% for BP.
- Black-box prompt tuning with OPS-LR reaches 65.1% on ImageNet versus 62.5% for LR and 57.9% for manual prompt.
- Multimodal alignment with OPS-LR reportedly reduces training time by almost half relative to BP, with allocator overhead negligible in wall time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use optimal query allocation with LR instead of equal allocation with LR | ViT-base · ImageNet | accuracy (%): 57.3 -> 65.8 | Helped |
| Use optimal query allocation with SPSA instead of equal allocation with DeepZero | ViT-base · ImageNet | accuracy (%): 55.6 -> 60.8 | Helped |
| Use optimal query allocation with LR instead of equal allocation with LR | ViT-base · Caltech101 | accuracy (%): 85.5 -> 88.9 | Helped |
| Use optimal query allocation with LR instead of equal allocation with LR | ViT-base · Food101 | accuracy (%): 87.6 -> 88.4 | Helped |
| Use optimal query allocation with LR instead of equal allocation with LR | ViT-base · Flowers102 | accuracy (%): 86.5 -> 91.6 | Helped |
| Use optimal query allocation with LR instead of equal allocation with LR | ViT-base · CIFAR10 | accuracy (%): 87.7 -> 93.2 | Helped |
| Use optimal query allocation with LR instead of equal allocation with LR | ViT-base · CIFAR100 | accuracy (%): 80.8 -> 88.7 | Helped |
| Use optimal query allocation with LR instead of equal allocation with LR | ViT-base · EuroSAT | accuracy (%): 82.9 -> 89.3 | Helped |
| Use optimal query allocation rather than equal allocation (ablation) | Forward learning; ViT-base on CIFAR100 · CIFAR100 | gradient cosine similarity | Helped |
| Use the Gaussian allocator instead of the Bernoulli allocator (ablation) | Forward learning; ViT-base on CIFAR100 · CIFAR100 | gradient cosine similarity | Helped |
| Use a correlated Gaussian allocation instead of a diagonal covariance allocation (ablation) | Gaussian allocator · CIFAR100 | gradient cosine similarity | Helped |
| Increase the query budget per data point (ablation) | Forward-learning gradient estimators · CIFAR100 | gradient cosine similarity | Helped |
Surprising findings
- OPS-LR on ViT-base CIFAR100 (88.7%) is nearly as accurate as BP but substantially more robust under natural and adversarial perturbations than BP in the robustness table.
- Mezo with only 2 queries per data point performs very poorly (e.g., 9.3% on ImageNet), indicating insufficient queries destabilize perturbation-based training.
Limitations
- Closing the accuracy gap with backpropagation remains open; OPS-LR still trails BP on clean ImageNet (65.8 vs 80.5).
- Memory cost of many queries is high for attention modules (O(n^2)); efficient attention integration left for future work.
- Gradient estimation quality for multi-head attention in deep layers is low, which may limit performance on large datasets.
Experimental setup
Datasets: ImageNet Caltech101 Food101 Flowers102 CIFAR10 CIFAR100 EuroSAT LCS-558K
Benchmarks: ImageNet Caltech101 Food101 Flowers102 CIFAR10/100 EuroSAT
Metrics: classification accuracy cosine similarity between estimated and true gradients wall-clock time
Baselines: BP MeZO DeepZero LR SPSA Manual Prompt
Builds on: SPSA MeZO DeepZero likelihood ratio gradient estimation Gaussian process / RBF kernel
Closest papers by meaning
- Forward Learning with Top-Down Feedback: Empirical and Analytical Characterization (ICLR 2024 · similarity 0.85)
- Efficient Backpropagation with Variance Controlled Adaptive Sampling (ICLR 2024 · similarity 0.85)
- Does This Gradient Spark Joy? (NeurIPS 2026 · similarity 0.85)
- Stepping Forward on the Last Mile (NeurIPS 2024 · similarity 0.85)
- Stable Coresets via Posterior Sampling: Aligning Induced and Full Loss Landscapes (NeurIPS 2025 · similarity 0.84)
- The Price of Locality: Why Forward-Forward Underperforms Backpropagation? (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.