Research map / Continual learning / Online and replay-based
FIRE: Frobenius-Isometry Reinitialization for Balancing the Stability–Plasticity Tradeoff
Isaac Han, Sangyeon Park, Seungwon Oh, Donghu Kim, Hojoon Lee, KyungJoong Kim
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use 1 Newton-Schulz iteration instead of 10 (ablation) | FIRE · Tiny ImageNet with VGG-16 (warm-start) | test accuracy | No clear effect |
| use Muon's tuned NS coefficients instead of (2, -1.5, 0.5) (ablation) | FIRE · CIFAR-10 ResNet-18 warm-start | test accuracy | No clear effect |
| reinitialize with FIRE instead of Shrink and Perturb in LLM continual pretraining | GPT-0.1B · OpenWebText + WikiText-103 | validation loss | Improved |
| full reset instead of keeping base model | GPT-0.1B · OpenWebText + WikiText-103 | validation loss | Hurt |
| FIRE instead of Muon optimizer (NS on gradients) | ResNet-18 · CIFAR-10 continual visual learning | test accuracy | Improved |
| FIRE instead of DASH on ViT-Tiny warm-start | ViT-Tiny · CIFAR-100 warm-start | test accuracy | Hurt |
| FIRE instead of S&P in DQN | DQN · Atari (Asterix, BeamRider, DemonAttack) | return | Improved |
| Plasticity Injection instead of FIRE | SAC/DQN · Atari and HumanoidBench | return | Hurt |
| orthogonalize MLP/output projections in ViT in addition to Q and K (ablation) | FIRE · ViT-Tiny | accuracy | No clear effect |
| FIRE instead of full reset / DASH in continual setting | ResNet-18, VGG-16 · continual visual learning | test accuracy | Improved |
Closest papers by meaning
- Self-Normalized Resets for Plasticity in Continual Learning (ICLR 2025 · similarity 0.85)
- Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspective (ICML 2025 · similarity 0.85)
- Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks (ICML 2024 · similarity 0.84)
- A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning (NeurIPS 2024 · similarity 0.83)
- Improving equilibrium propagation without weight symmetry through Jacobian homeostasis (ICLR 2024 · similarity 0.83)
- Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
