Research map / Reinforcement learning / RL from feedback
Reducing the Probability of Undesirable Outputs in Language Models Using Probabilistic Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RePULSe, which augments the standard RL loss (REINFORCE/RLOO) with a loss that samples low-reward outputs via learned proposals and importance sampling from a tracking target distribution, then reduces their probability to improve the reward vs. undesirable-output tradeoff.
Problem
Standard RL alignment of language models reduces the probability of low-reward (undesirable) outputs slowly because such outputs are rarely sampled once the model improves; methods that push harder on them usually hurt average reward.
Approach
Build a target distribution that concentrates on low-reward sequences under the current LM, learn an autoregressive proposal q_xi (initialized from p_theta) by mass-covering CTL/twist learning to sample from it via self-normalized importance sampling, and add a gradient-ascent (negative SFT) loss on those samples to the standard REINFORCE loss.
Key results
- On the SmolLM-135M setting, RePULSe achieves a better Pareto frontier of average return versus probability of bad outputs than REINFORCE baselines, while using half as many policy gradient updates.
- In the toy DistilGPT2 experiment, RePULSe monotonically reduces total probability of bad outputs to a much lower level than RLOO and PPO with nearly identical average reward.
- RePULSe reduces GCG attack success rate compared to baselines on 10 held-out prompts, suggesting improved adversarial robustness.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the RePULSe low-reward sampling and probability-reduction loss | DistilGPT2 alignment · Toy experiment (single prompt) | probability of bad outputs | Helped |
| Add the RePULSe low-reward sampling and probability-reduction loss | DistilGPT2 alignment · Toy experiment (single prompt) | average reward | No clear effect |
| Add learned proposal sampling and low-reward probability reduction | SmolLM-135M-Instruct alignment · Held-out prompts; reward threshold -5 | reward-versus-bad-output probability frontier | Helped |
| Add learned proposal sampling and low-reward probability reduction | Llama-3.2-1B-Instruct alignment · Held-out prompts; reward threshold -7 | reward-versus-bad-output probability frontier | Helped |
| Add learned proposal sampling and low-reward probability reduction | Language model alignment · GCG adversarial attack; SmolLM-135M-Instruct and Llama-3.2-1B-Instruct | GCG attack success rate | Helped |
| Use the policy p_theta itself instead of the learned proposal q_xi (ablation) | RePULSe · More realistic experiments; reward-versus-bad-output probability frontier | reward-versus-bad-output probability frontier | Hurt |
| Use reward-thresholding target p_theta I[r < eta] instead of negative-temperature scaling (ablation) | RePULSe · More realistic experiments | performance | Hurt |
| Train the proposal with DPG instead of CTL (ablation) | RePULSe proposal learning · SmolLM setting | reward-versus-bad-output probability tradeoff | No clear effect |
| Train the proposal with DPG instead of CTL (ablation) | RePULSe proposal learning · Llama setting | reward-versus-bad-output probability tradeoff | Hurt |
| Use a REINFORCE loss on low-reward target samples instead of directly reducing their log probability (ablation) | RePULSe · Low-reward target sampling experiment | undesirable-output performance | Hurt |
| Train RePULSe for fewer policy updates under lower KL penalties (ablation) | RePULSe · Lower-KL-penalty settings | reward-versus-bad-output probability tradeoff | Mixed |
Surprising findings
- Using REINFORCE on samples from the low-reward target without a high baseline performs poorly, as it increases probability of bad-but-not-the-worst sequences.
- DPG as a proposal learner performed worse than CTL on the Llama setting due to a feedback loop when all proposal samples are identical.
Limitations
- Results depend on being close to convergence; with lower KL penalties or limited training, halving policy updates may hurt RePULSe.
- Small-scale models (135M, 1B) and short outputs limit evidence for frontier-scale models.
- Proposal q may need constant exploration; a better-trained base model gives little signal for learning q.
Experimental setup
Datasets: Adversarial prompts (cited source) OpenRLHF prompt collection (includes UltraFeedback)
Benchmarks: GCG adversarial attack success rate
Metrics: average return including KL penalty probability of bad outputs (reward below threshold) CVaR GCG attack success rate
Baselines: RLOO PPO reward-transformed REINFORCE p_theta-proposal ablation
Builds on: REINFORCE leave-one-out (RLOO) Contrastive Twist Learning Distributional Policy Gradient Greedy Coordinate Gradient
Closest papers by meaning
- Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback (AAAI 2025 · similarity 0.89)
- ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models (ICML 2024 · similarity 0.89)
- Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.89)
- Theoretical Limits of Language Model Alignment (NeurIPS 2026 · similarity 0.89)
- RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style (ICLR 2025 · similarity 0.89)
- Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.