Research map / Large language models / Language, safety and interpretability
QUARK: Controllable Text Generation with Reinforced Unlearning
Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, Yejin Choi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Quark instead of PPO | GPT2-large · RealToxicityPrompts | avg. max. toxicity: 0.218 -> 0.196 | Improved |
| Use Quark instead of PPO | GPT2-large · RealToxicityPrompts | toxic probability: 0.044 -> 0.035 | Improved |
| Use Quark instead of PPO | GPT2-large · Sentiment unlearn negative | % positive: 43.13 -> 46.55 | Improved |
| Use Quark instead of PPO | GPT2-large · Sentiment unlearn positive | % positive: 32.22 -> 27.50 | Improved |
| Add unlikelihood loss to Quark | GPT2-base · Wikitext-103 | rep-2: 39.89 -> 18.76 | Improved |
| Add unlikelihood loss to Quark | GPT2-base · Wikitext-103 | MAUVE: 0.74 -> 0.82 | Improved |
| Add unlikelihood loss to Quark | GPT2-base · Wikitext-103 | perplexity: 26.22 -> 27.97 | Hurt |
| Exact token-level KL vs approximate point-wise KL (ablation) | Quark · RealToxicityPrompts val | output perplexity: 13.86 -> 12.72 | Improved |
| Add exact KL term (vs none) (ablation) | Quark · RealToxicityPrompts val | output perplexity: 13.29 -> 12.72 | Improved |
| Increase KL coefficient beta (ablation) | Quark · RealToxicityPrompts val | output perplexity | Mixed |
| Increase number of quantiles (ablation) | Quark · RealToxicityPrompts val | toxicity | Mixed |
| Train only on best quantile instead of all (ablation) | Quark · RealToxicityPrompts val | output perplexity: 12.72 -> 21.92 | Hurt |
Later papers that compared with this method
1 later accepted paper reports Quark as a baseline in their results tables; 1 of them report a better number on most of the results they share with it.
- Risk-Averse Fine-tuning of Large Language Models (NeurIPS 2024 · better on 1 of 1 shared result)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Towards Cost-Effective Reward Guided Text Generation (ICML 2025 · similarity 0.88)
- Reducing the Probability of Undesirable Outputs in Language Models Using Probabilistic Inference (NeurIPS 2025 · similarity 0.88)
- Vanishing Gradients in Reinforcement Finetuning of Language Models (ICLR 2024 · similarity 0.88)
- GRAM: A Generative Foundation Reward Model for Reward Generalization (ICML 2025 · similarity 0.88)
- Inference-Time Reward Hacking in Large Language Models (NeurIPS 2025 · similarity 0.87)
- GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-Time Alignment (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
