Research map / Label-efficient and robust learning / Recognition and long-tail
C$^{2}$R: Cross-sample Consistency Regularization Mitigates Feature Splitting and Absorption in Sparse Autoencoders
Haoran Jin, Xiting Wang, Shijie Ren, Hong Xie, Defu Lian
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add C2R regularization to Batch TopK SAE | Batch TopK SAE · Gemma-2-2B layer 12, SAEBench | feature absorption and splitting | Improved |
| add C2R regularization to Batch TopK SAE | Batch TopK SAE · Gemma-2-2B layer 12, SAEBench | reconstruction fidelity (loss recovered, KL score) | No clear effect |
| add C2R regularization to Batch TopK SAE | Batch TopK SAE · Gemma-2-2B layer 12, AutoInterp | AutoInterp score | No clear effect |
| add C2R regularization to Batch TopK SAE | Batch TopK SAE · RAVEL disentanglement | disentanglement score | No clear effect |
| add C2R regularization to Batch TopK SAE | Batch TopK SAE · Gemma-2-2B layer 12 | feature composition (max cosine similarity) | Improved |
| use Matryoshka SAE instead of C2R-regularized Batch TopK | SAE architectures · Gemma-2-2B layer 12 | reconstruction fidelity | Hurt |
| remove nearest-neighbor restriction (penalize all latent pairs) (ablation) | C2R · Gemma-2-2B layer 12, k=100 | reconstruction fidelity and composition | Hurt |
| remove ReLU cosine gate (ablation) | C2R · Gemma-2-2B layer 12, k=100 | reconstruction fidelity and composition | Hurt |
| increase lambda_C2R from 0.1 up to 5 (ablation) | C2R · Gemma-2-2B layer 12, k=100 | composition and absorption | Improved |
| increase lambda_C2R to 10 (ablation) | C2R · Gemma-2-2B layer 12, k=100 | reconstruction fidelity | Hurt |
| apply C2R on top of TopK, OrtSAE and AbsTopK backbones | SAE backbones · Gemma-2-2B layer 12, k=100 | composition and absorption | Improved |
| apply C2R to larger models (Qwen3-8B, Llama-3-8B layer 20) | Batch TopK SAE · Qwen3-8B and Llama-3-8B, k=100 | composition and absorption | Improved |
Closest papers by meaning
- Scaling and evaluating sparse autoencoders (ICLR 2025 · similarity 0.90)
- A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders (NeurIPS 2025 · similarity 0.90)
- SASA: Subspace-Aware Sparse Autoencoders for Effective Mechanistic Interpretability (NeurIPS 2026 · similarity 0.89)
- Are Sparse Autoencoders Useful? A Case Study in Sparse Probing (ICML 2025 · similarity 0.89)
- SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability (ICML 2025 · similarity 0.88)
- Semantic Optimal Transport for Sparse Autoencoder Feature Matching and Circuit Compression (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
