Research map / Probabilistic and scientific modeling / Molecules and proteins
Learning a Generative Meta-Model of LLM Activations
Grace Luo, Jiahai Feng, Trevor Darrell, Alec Radford, Jacob Steinhardt
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| scale GLP from 3 to 24 layers (0.5B to 3.3B) (ablation) | GLP on Llama1B · Frechet distance | FD: 0.68 -> 0.53 | Improved |
| use GLP instead of SAE reconstruction | Llama8B activations · Frechet distance | FD: 6.91 -> 5.93 | Improved |
| use GLP instead of SAE reconstruction | Llama8B-Base · Delta LM Loss | delta LM loss: 0.1976 -> 0.0513 | Improved |
| transfer GLP trained on Base to Instruct (ablation) | GLP · Delta LM Loss | delta LM loss: 0.0513 -> 0.0860 | Hurt |
| use GLP meta-neurons instead of raw MLP neurons | Llama8B · 1-D probing | probe AUC: 0.82 -> 0.87 | Improved |
| use GLP meta-neurons instead of SAE | Llama1B · 1-D probing | probe AUC: 0.70 -> 0.84 | Improved |
| use GLP features instead of raw MLP neurons | Llama8B · Dense probing | probe AUC: 0.94 -> 0.94 | No clear effect |
| use GLP instead of SAE | Llama1B · Dense probing | probe AUC: 0.85 -> 0.92 | Improved |
| train one multi-layer GLP on all layers instead of single-layer (ablation) | GLP on Llama1B · Frechet distance | FD: 0.55 -> 0.66 | Hurt |
| post-process steering with GLP diffusion projection (t_start 0.5, 20 steps) | DiffMean, Persona Vector, SAE steering · Steering concept vs fluency | Pareto frontier | Improved |
| scale GLP compute (ablation) | GLP on Llama1B · Diffusion loss | loss power law: irreducible error 0.52 | Improved |
| prefilter top-512 candidate features before 1-D probing (ablation) | Raw layer output / MLP neuron · 1-D probing | probe AUC: 0.82 -> 0.82 | No clear effect |
Closest papers by meaning
- CASL: Concept-Aligned Sparse Latents for Interpreting Diffusion Models (NeurIPS 2026 · similarity 0.87)
- Controlling Language and Diffusion Models by Transporting Activations (ICLR 2025 · similarity 0.87)
- Hyper-Transforming Latent Diffusion Models (ICML 2025 · similarity 0.86)
- Why DiT Models Underperform as Representation Learners without Long Skip Connections (NeurIPS 2026 · similarity 0.86)
- Beyond Linear Activation Steering: Invertible Latent Transformations for Controlling LLM Behavior (NeurIPS 2026 · similarity 0.86)
- Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
