AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

Breaking the Block: Preserving Data Continuity to Train Superior SAEs for Instruct Models

Jiaming Li, Haoran Ye, Yukun Chen, Xinyue Li, Lei Zhang, Hamid Alinejad-Rokny, Jimmy Chih-Hsien Peng, Min Yang

ICML 2026 · Architectures and efficiency › RNNs and normalization · approach: Sparse dictionary representations · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
train SAE sequentially on independent instances (FAST) instead of concatenated blocksSAE training on instruct models · Qwen2.5-7B-Instruct L18 8X Standardlog2 MSEst: 5.1985 -> 0.6468Improved
use FAST instead of block trainingJumpReLU SAE · special-token reconstructionlog2 MSEst: -4.0005 -> -9.7604Improved
use FAST instead of block trainingSAE training · Delta LossDelta Loss: -0.51% to 0.37% vs deviations up to 51.43%Improved
use FAST instead of block trainingSAE training · Llama-3.1-8B early trainingGSNR: 0.26 -> 0.50Improved
use FAST instead of block trainingSAE feature quality · Llama3.2-3B-Instructhigh-quality features %: 7.0% / 10.2% -> 21.1%Improved
limit FAST truncation to 2048 tokens to match block context (ablation)FAST · Llama-3.1-8B-Instruct L20 8X JumpReLUlog2(MSE): -15.56 -> -15.42No clear effect
match context length at 2048 tokens, FAST vs BT(F) (ablation)FAST · Llama-3.1-8B-Instruct L20 8X JumpReLUKL divergence: 8.19e-05 -> 2.62e-05Improved
prepend a random 512-token unrelated prefix to each FAST instance (RandomConcat) (ablation)FAST · early training Steps 0-100GSNRHurt
steer special-token features with moderate scaling (alpha 15-50)FAST JumpReLU SAE 8X · 1,010 instruction instancesgeneration qualityImproved
steer special-token features with excessive scaling (alpha > 100) (ablation)FAST JumpReLU SAE 8X · 1,010 instruction instancesgeneration qualityHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy