Research map / Architectures and efficiency / RNNs and normalization
PolySAE: Modeling Feature Interactions in Sparse Autoencoders via Polynomial Decoding
Panagiotis Koromilas, Andreas D. Demou, James Oldfield, Yannis Panagakis, Mihalis Nicolaou
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add polynomial (quadratic and cubic) decoder terms | TopK SAE on GPT-2 Small · SAEBench sparse probing K=1 (GPT-2 Small) | mean F1 (%): 67.1 -> 77.9 | Improved |
| add polynomial (quadratic and cubic) decoder terms | TopK SAE on Pythia-410M · SAEBench sparse probing K=1 (Pythia-410M) | mean F1 (%): 71.2 -> 77.0 | Improved |
| add polynomial (quadratic and cubic) decoder terms | TopK SAE on Pythia-1.4B · SAEBench sparse probing K=1 (Pythia-1.4B) | mean F1 (%): 75.9 -> 81.9 | Improved |
| add polynomial (quadratic and cubic) decoder terms | TopK SAE on Gemma-2-2B · SAEBench sparse probing K=1 (Gemma-2-2B) | mean F1 (%): 67.7 -> 68.4 | Improved |
| add polynomial (quadratic and cubic) decoder terms | BatchTopK SAE on GPT-2 Small · SAEBench sparse probing K=1 (GPT-2 Small) | mean F1 (%): 65.7 -> 78.0 | Improved |
| add polynomial (quadratic and cubic) decoder terms | BatchTopK SAE on Pythia-410M · SAEBench sparse probing K=1 (Pythia-410M) | mean F1 (%): 65.0 -> 77.3 | Improved |
| add polynomial (quadratic and cubic) decoder terms | BatchTopK SAE on Pythia-1.4B · SAEBench sparse probing K=1 (Pythia-1.4B) | mean F1 (%): 64.6 -> 76.4 | Improved |
| add polynomial (quadratic and cubic) decoder terms | BatchTopK SAE on Gemma-2-2B · SAEBench sparse probing K=1 (Gemma-2-2B) | mean F1 (%): 64.8 -> 69.4 | Improved |
| add polynomial (quadratic and cubic) decoder terms | Matryoshka SAE on GPT-2 Small · SAEBench sparse probing K=1 (GPT-2 Small) | mean F1 (%): 65.7 -> 77.7 | Improved |
| add polynomial (quadratic and cubic) decoder terms | Matryoshka SAE on Gemma-2-2B · SAEBench sparse probing K=1 (Gemma-2-2B) | mean F1 (%): 60.9 -> 65.6 | Improved |
| add low-rank factorization to the polynomial decoder (ablation) | polynomial decoder with shared projection · Architectural ablation (GPT-2 Small) | mean F1 (%): 76.0 -> 75.0 | Mixed |
| add orthogonality constraint on the projection (ablation) | low-rank polynomial decoder · Architectural ablation (GPT-2 Small) | mean F1 (%): 75.0 -> 77.9 | Improved |
Closest papers by meaning
- SASA: Subspace-Aware Sparse Autoencoders for Effective Mechanistic Interpretability (NeurIPS 2026 · similarity 0.90)
- Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry (NeurIPS 2025 · similarity 0.89)
- Semantic Optimal Transport for Sparse Autoencoder Feature Matching and Circuit Compression (NeurIPS 2026 · similarity 0.88)
- Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment (ICML 2025 · similarity 0.88)
- A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders (NeurIPS 2025 · similarity 0.88)
- Scaling and evaluating sparse autoencoders (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
