Research map / Large language models / Language, safety and interpretability
Self-conditioning Pre-Trained Language Models
Xavier Suau Cuadros, Luca Zappella, Nicholas Apostoloff
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use expert-unit intervention instead of FUDGE | GPT2-M gender parity generation · Occupational contexts, woman parity | perplexity: 85.40, (83.80, 92.49) -> 69.50, (63.43, 71.72) | Improved |
| Use expert-unit intervention instead of FUDGE | GPT2-M gender parity generation · Occupational contexts, man parity | perplexity: 83.29, (81.60, 129.09) -> 64.62, (63.54, 65.02) | Improved |
| Use expert-unit intervention instead of PPLM-BoW | GPT2-M gender parity generation · Occupational contexts, woman parity | perplexity: 288.19,(202.16, 502.35) -> 69.50, (63.43, 71.72) | Improved |
| Use expert-unit intervention instead of PPLM-BoW | GPT2-M gender parity generation · Occupational contexts, man parity | perplexity: 262.48, (153.82, 411.49) -> 64.62, (63.54, 65.02) | Improved |
| Intervene on top-ranked expert units rather than lower-ranked or random units (ablation) | Self-conditioning · Nurse and doctor contexts | probability of generating the target gender concept | Improved |
| Increase the number of intervened expert units (ablation) | GPT2-L bird-conditioned generation · Bird concept | presence of the conditioned concept | Improved |
| Set expert units to zero to remove a concept (ablation) | Self-conditioning · Concept removal | concept removal effectiveness | No clear effect |
| Use more expert units for a weakly detected concept (ablation) | GPT2-L self-conditioned generation · Work concept | generation quality | Hurt |
| Use the self-conditioning intervention rather than PPLM-BoW | GPT2 inference · Generation speed | speedup: 7.3× | Improved |
| Use self-conditioning rather than FUDGE or PPLM-BoW at strong gender conditioning | Gender parity generation · Occupational contexts | probability of generating the literal conditioning word | Improved |
Closest papers by meaning
- SCOPE: A Self-supervised Framework for Improving Faithfulness in Conditional Text Generation (ICLR 2025 · similarity 0.87)
- Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass (ICLR 2025 · similarity 0.87)
- It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs (NeurIPS 2026 · similarity 0.86)
- From Self-Attention to Markov Models: Unveiling the Dynamics of Generative Transformers (ICML 2024 · similarity 0.86)
- Precise Information Control in Long-Form Text Generation (NeurIPS 2025 · similarity 0.86)
- Position: Do pretrained Transformers Learn In-Context by Gradient Descent? (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
