Research map / Optimization / Optimizers
Fishers for Free? Approximating the Fisher Information Matrix by Recycling the Squared Gradient Accumulator
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the "Squisher", which reuses Adam's squared gradient accumulator as a free approximation of the empirical Fisher diagonal, and shows it performs comparably to the Fisher across five applications (merging, pruning, FISH mask, task embedding, EWC).
Problem
Computing the Fisher diagonal, used as a parameter-sensitivity measure, requires extra per-example gradient computation and training data access, which is costly and inconvenient; the question is whether the squared gradient accumulator already maintained by Adam can replace it.
Approach
Relate the squared gradient accumulator (EMA of squared mini-batch averaged gradients) to the joint empirical Fisher diagonal, which is a square of a sum rather than a sum of squares, so the accumulator can be recycled as a Fisher proxy, with rescaling by N where magnitude matters.
Key results
- Fisher merging of T5 models: Squisher averages 58.0% vs Fisher 53.6% and linear averaging 48.5%.
- Continual learning Split CIFAR-100 task-incremental: Squisher 75.30% vs Fisher 75.96% vs no regularization 61.82%.
- FISH mask on GLUE with BERT-Large (50% sparsity): Squisher average 82.3 vs Fisher 82.4 vs random 73.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the Fisher with the squared gradient accumulator in Fisher merging | T5-Large-LM-Adapt model merging · Fisher Merging (Average) | accuracy: 53.6 -> 58.0 | Helped |
| Replace the Fisher with the squared gradient accumulator in UBGM | RoBERTa model merging · UBGM (Average) | accuracy: 93.99 -> 94.00 | No clear effect |
| Use the squared gradient accumulator to rank parameters for pruning | VGG-13 Fisher pruning · Fisher Pruning (75% pruned) | accuracy: 64.8 -> 63.9 | Hurt |
| Use the squared gradient accumulator to select parameters | BERT-Large FISH Mask · FISH Mask (AVG) | accuracy: 82.4 -> 82.3 | No clear effect |
| Use optimizer squared-gradient accumulators for task embeddings | BERT Task Embedding · classification/regression in-class (10) | MRR: 0.421 -> 0.468 | Helped |
| Use optimizer squared-gradient accumulators for task embeddings | BERT Task Embedding · question answering in-class (10) | MRR: 0.460 -> 0.563 | Helped |
| Replace Fisher in the EWC regularizer with the scaled squared gradient accumulator | EWC · Split CIFAR-100 task-incremental | accuracy: 75.96 -> 75.30 | No clear effect |
| Remove Squisher rescaling and use the Fisher-default lambda (ablation) | EWC · Split CIFAR-100 task-incremental | accuracy: 75.30 -> 61.53 | Hurt |
| Set Adam β₂ to 0.95 (ablation) | Squisher in EWC · Split CIFAR-100 task-incremental | accuracy: 75.30 -> 72.80 | Hurt |
| Turn off the moving average and explicitly square the sum of per-example gradients (ablation) | Squisher in EWC · Split CIFAR-100 task-incremental | accuracy: 75.30 -> 75.38 | No clear effect |
Surprising findings
- Squisher merging performance degrades when fine-tuned models are not fully trained, and the Fisher-vs-Squisher ranking flips depending on whether final or best checkpoints are used.
- Without rescaling by N, the Squisher performs substantially worse in continual learning (e.g., Split CIFAR-100 61.53% vs 75.30% with rescaling).
Limitations
- Results show inconsistent winner between Fisher and Squisher, depending on training duration and checkpoint choice.
- Squisher requires rescaling by N for magnitude-sensitive uses like EWC, adding a tuning heuristic without theoretical justification.
- Only diagonal approximations are studied; non-diagonal Fisher not covered.
Experimental setup
Datasets: cosmos_qa paws qasc quail quartz ropes social_iqa wiki_qa IMDB Yelp Rotten Tomatoes SST2
Benchmarks: Fisher merging multitask QA suite UBGM RoBERTa sentiment merging VGG-13 CIFAR-100 pruning GLUE with BERT-Large FISH mask Task2Vec transferability ranking (22 datasets) Split MNIST Permuted MNIST Split CIFAR-100 continual learning
Metrics: average accuracy mean reciprocal rank NDCG average rank
Baselines: Fisher diagonal linear parameter averaging random pruning mask no regularization EWC random FISH mask dataset size heuristic
Builds on: Fisher merging Elastic Weight Consolidation FISH Mask Task2Vec Adam joint Fisher
Closest papers by meaning
- Trade-Offs of Diagonal Fisher Information Matrix Estimators (NeurIPS 2024 · similarity 0.86)
- An Improved Empirical Fisher Approximation for Natural Gradient Descent (NeurIPS 2024 · similarity 0.86)
- AdaFisher: Adaptive Second Order Optimization via Fisher Information (ICLR 2025 · similarity 0.85)
- Direct Fisher Score Estimation for Likelihood Maximization (NeurIPS 2025 · similarity 0.85)
- Prodigy: An Expeditiously Adaptive Parameter-Free Learner (ICML 2024 · similarity 0.84)
- Bayesian Optimization with Fisher Information Geometry: Gradient Bounds and Trust-Region Methods (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.