Research map / Large language models / Language, safety and interpretability
Successor Features for Efficient Multi-Subject Controlled Text Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SFGen, which applies successor features with language model rectification to controllable text generation, learning one feature network plus a small reward vector per subject so multiple subjects can be combined at decoding time without altering the LLM.
Problem
Decoding-based controllable text generation needs new discriminators whenever the target subject changes, and becomes costly when multiple subjects (e.g., several toxicity attributes or sentiments) must be controlled simultaneously.
Approach
Combine successor features (decoupling LM dynamics from task reward via a linear reward w) with language model rectification; reformulate SF Bellman updates so the linear reward is only regressed at terminal states, and fuse multiple subjects by taking the minimum of their value functions.
Key results
- On RealToxicityPrompts, the method reaches expected max toxicity 0.35 versus 0.30 for Rect and 0.38 for DExperts, with output perplexity 48.17 versus 52.80 for Rect.
- On the 6B GPT4All-J model, the method reduces toxicity to 0.34 from 0.69 for the unprompted model and 0.56 for direct instruction prompting.
- Sentiment control: the method reaches 46.78% positive for negative prompts, behind Rect (52.02%) and DExperts (31.64%, lower than ours on negative-prompt cases), and is slightly behind Rect overall.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SFGen instead of direct prompting for detoxification | GPT4All-J 6B · Detoxification, 6B instruction-tuned LM | toxicity: 0.56_{0.13} -> 0.34_{0.15} | Helped |
| Use SFGen instead of direct prompting for detoxification | GPT4All-J 6B · Detoxification, 6B instruction-tuned LM | insult: 0.46_{0.18} -> 0.19_{0.14} | Helped |
| Use SFGen instead of direct prompting for detoxification | GPT4All-J 6B · Detoxification, 6B instruction-tuned LM | threat: 0.14_{0.17} -> 0.08_{0.10} | Helped |
| Use SFGen instead of direct prompting for detoxification | GPT4All-J 6B · Detoxification, 6B instruction-tuned LM | sexual toxicity: 0.32_{0.25} -> 0.18_{0.18} | Helped |
| Use SFGen instead of Rect | GPT-2 large · Detoxification, RealToxicityPrompts toxic prompts | output perplexity: 52.80 -> 48.17 | Helped |
| Use SFGen instead of Rect | GPT-2 large · Sentiment control, negative prompts | positive sentiment %: 52.02 -> 46.78 | Hurt |
| Use SFGen instead of Rect | GPT-2 large · Sentiment control, positive prompts | negative sentiment %: 74.20 -> 70.29 | Hurt |
| Combine attack, threat, and sexual reward parameters | SFGen · Detoxification, 500-prompt subset | attack: 0.50_{0.15} -> 0.24_{0.18} | Helped |
| Combine attack, threat, and sexual reward parameters | SFGen · Detoxification, 500-prompt subset | threat: 0.48_{0.15} -> 0.13_{0.13} | Helped |
| Combine attack, threat, and sexual reward parameters | SFGen · Detoxification, 500-prompt subset | sexual toxicity: 0.68_{0.18} -> 0.34_{0.18} | Helped |
| Train successor features with Monte Carlo rather than SARSA | SFGen · Sentiment control and detoxification | performance | No clear effect |
Surprising findings
- Direct instruction prompting of a 6B instruction-tuned LM only modestly reduces toxicity (0.69 to 0.56), far less than the decoding-based method.
- The authors' re-run of baselines on Perspective API gave lower toxicity scores than previously reported numbers for the same baselines.
Limitations
- Linear reward constraint makes it slightly behind Rect on sentiment control.
- Relies on imperfect toxicity classifiers (Perspective API) as labels and evaluators, with ethical concerns about false positives and negatives.
Experimental setup
Datasets: RealToxicityPrompts OpenWebText (OWT) corpus SST-2
Benchmarks: RealToxicityPrompts OpenWebText sentiment prompts
Metrics: toxicity expected max score (Perspective API) positive/negative sentiment percentage output perplexity (GPT2-XL) Dist-2 Dist-3 inference time
Baselines: PPLM DAPT CTRL GeDi DExperts Rect Self-Debiasing
Builds on: successor features language model rectification DExperts GeDi PPLM
Closest papers by meaning
- Controlled Text Generation via Language Model Arithmetic (ICLR 2024 · similarity 0.88)
- Precise Information Control in Long-Form Text Generation (NeurIPS 2025 · similarity 0.86)
- Decoding Game: On Minimax Optimality of Heuristic Text Generation Strategies (ICLR 2025 · similarity 0.86)
- Controlling Large Language Models Through Concept Activation Vectors (AAAI 2025 · similarity 0.85)
- Improving Open-Ended Text Generation via Adaptive Decoding (ICML 2024 · similarity 0.85)
- Test-Time Alignment for Large Language Models via Textual Model Predictive Control (ICLR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.