AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

Scaling Laws for SAE Training Data

Nikita Koryagin, Nikita Balagansky, Daniil Gavrilov

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Sparse dictionary representations · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that sparse autoencoder training quality depends mainly on the number of unique activations rather than fresh tokens, so small replayed activation buffers can match much larger training budgets.

Problem

SAE training is often bottlenecked by activation storage, and it is unclear how much stored activation data is actually needed relative to the total training budget.

Approach

Quality depends on the count of unique activations rather than whether every training token is fresh; after a short diversity-limited regime, replaying a fixed buffer preserves most quality, so the buffer can be far smaller than the training budget.

Key results

Measured effects reported in the paper

What was changedWhereMeasuredEffect
increase unique activations from 1M to 128M under a fixed 128M budget (ablation)BatchTopK SAE 16k · 128M budgetexplained variance: 0.5801 -> 0.6393Improved
increase unique activations from 1M to 128M (ablation)BatchTopK SAE 32k · 128M budgetexplained variance: 0.5870 -> 0.6610Improved
increase unique activations beyond the plateau (replay instead of fresh data) (ablation)BatchTopK SAE · fixed budgetexplained variance: within 0.005 of maxNo clear effect
double dictionary size (ablation)BatchTopK SAE · 1.28B tokenssaturation interval: 20M-40M -> 40M-80MHurt
replace BatchTopK with TopK or JumpReLU (ablation)SAE · D=32k, 640Msaturation point: 40M for allNo clear effect
math text instead of FineWeb (ablation)SAE · FineMath-3plussaturation buffer: 80M, 8x reductionMixed
Python code instead of FineWeb (ablation)SAE · Stack-Edu Pythonsaturation buffer: 40M, 16x reductionMixed
tighten EV tolerance from 0.005 to 0.0025 (ablation)SAE · Stack-Edu Pythonstorage reduction: 16x -> 8xHurt
let repetition scale depend on dictionary size (ablation)scaling law · 96-run gridAIC: delta AIC -77.3Improved
increase buffer beyond EV-chosen size (ablation)SAE · SAEBenchAutoInterp, sparse probingNo clear effect
increase unique activations (ablation)SAE 32k and 64k · SAEBenchabsorption fractionHurt

Surprising findings

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy