AI Paper Atlas

Research map / Large language models / Instruction and fine-tuning

SAE as a Crystal Ball: Interpretable Features Predict Cross-domain Transferability of LLMs without Training

Qi Zhang, Yifei Wang, Xiaohan Wang, Jiajun Chai, Guojun Yin, Wei Lin, Yisen Wang

ICLR 2026 · Large language models › Instruction and fine-tuning · approach: Fine-tuning generalization and delta compression · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use STS_ICL instead of STS_act (activation-sum) to measure domain correlation with shifted dimensions (ablation)STS · MMLU-Pro (SFT on LIMO)Pearson correlation: Gemma2-9B 0.60 -> 0.77; Llama3-8B 0.71 -> 0.81; Qwen2.5-7B 0.90 -> 0.78Mixed
Use SAE features with higher sparsity (lower L0) (ablation)STS on Gemma2-9B-Instruct · MMLU-Pro (SFT on LIMO)Pearson correlation: L0 109: 0.65 -> L0 13: 0.78Improved
Use SAE with larger hidden dimension (131k vs 16k, stronger monosemanticity) (ablation)STS · MMLU-Pro (SFT on LIMO)Pearson correlationImproved
Select dimensions with the largest shifts rather than smaller-shifted dimensions (ablation)STS · MMLU-Pro (SFT on LIMO)Pearson correlationImproved
Apply SAE layer choice of 15, 20 or 25 instead of fixed layer (ablation)STS on Qwen2.5-7B-Instruct · MMLU-Pro (SFT on LIMO)Pearson correlationNo clear effect
Use SAE features instead of raw model features to find ICL-SFT shifted dimensions (ablation)shifted dimension identification · LIMO SFT of Qwen2.5-7B-Instructoverlap of top-100 shifted dims: 57% of the top 100 most shifted SAE dimensions coincideImproved
Use STS instead of raw feature activations as predictortransferability prediction · MMLU-Pro (Qwen SFT on LIMO)correlation coefficient: 0.03 -> 0.79Improved
Use STS instead of SAE feature activation probetransferability prediction · MMLU-Pro (Qwen SFT on LIMO)correlation coefficient: 0.08 -> 0.79Improved
Use STS (no SFT model needed) instead of representation similarity before/after SFTtransferability prediction · MMLU-Pro (Qwen SFT on LIMO)correlation coefficient: 0.61 -> 0.79Improved
Allocate STS-proportional extra data to engineering and law domainsSFT of Qwen2.5-7B-Instruct on LIMO · MMLU-Pro engineering and lawdownstream accuracyImproved
Use STS-guided data mixture instead of LIMO onlySFT of Qwen2.5-7B-Instruct · Math-LightEvalaccuracy (%): 81.3 -> 80.1No clear effect
Fine-tune estimated shifted dimensions instead of random dimensions (3000 dims) (ablation)selective fine-tuning · Math-LightEvalaccuracy (%): 74.0 -> 76.9Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy