Research map / Large language models / Instruction and fine-tuning

Risk-Averse Fine-tuning of Large Language Models

NeurIPS 2024 · Large language models › Instruction and fine-tuning · approach: Safety alignment after fine-tuning · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes risk-averse RLHF (RA-RLHF) that applies CVaR-style soft-risk scheduling of batch selection to PPO-based RLHF so GPT-2 and GPT-J generate less toxic or more positive continuations, especially for tail prompts.

Problem

Standard RLHF maximizes expected reward, so rare but harmful (toxic or negative) generations from hard prompts are not explicitly suppressed; safety-constrained RLHF constrains expected harmfulness rather than the probability of toxic tail outcomes.

Approach

Adapts CVaR risk-averse RL (soft-risk scheduling from CeSoR) to RLHF: in each PPO iteration, only the B0 lowest-return trajectories in a batch are used for updates, with B0 gradually shrinking from the full batch to ceil(alpha*B) following a schedule, while a SFT reference and adaptive KL penalty regularize the policy.

Key results

• On Jigsaw-Gen with GPT-2, RA-RLHF reaches -Tox 2.0568 versus 1.6933 for RLHF and 0.3480 for base GPT-2.
• On RealToxicityPrompts-Gen with GPT-2, RA-RLHF reaches -Tox 2.8335 versus 2.5612 for RLHF.
• On IMDB-Gen with GPT-J, RA-RLHF tail reward is -1.11 versus -1.51 for RLHF, with perplexity 23.03 versus 22.13.

Measured effects reported in the paper

Surprising findings

• RA-RLHF's perplexity is only marginally higher than RLHF, and in a GPT-J IMDB run RA-RLHF's tail reward gain comes with a slight perplexity increase.
• Too aggressive risk aversion (low alpha, high rho) trades reward for higher perplexity, and the policy does not collapse into repeating positive words like 'great great great'.

Limitations

• Simplified selection of the lowest-return B0 trajectories rather than the exact empirical CVaR quantile estimator.
• GPT-J experiments trained on only 70% of the training data due to a 24-hour GPU time limit and used a GPT-2 SFT model as reference.
• Evaluated only on generative sentiment and toxicity tasks, not Q&A or conversational settings; slight perplexity increase.

Experimental setup

Datasets: IMDB Jigsaw Toxic Comment (Kaggle 2017) RealToxicityPrompts

Benchmarks: IMDB-Gen Jigsaw-Gen RealToxicityPrompts-Gen

Metrics: tail average reward toxicity score (-Tox) perplexity Dist-1 Dist-2 Dist-3

Baselines: Base LLM Prompted base LLM DExperts SFT RLHF Quark

Builds on: CeSoR PPO RLHF DExperts Quark

Closest papers by meaning

• Rule Based Rewards for Language Model Safety (NeurIPS 2024 · similarity 0.87)
• Aligning Language Models with Selective Prediction (NeurIPS 2026 · similarity 0.87)
• Safe RLHF: Safe Reinforcement Learning from Human Feedback (ICLR 2024 · similarity 0.87)
• When Bad Data Leads to Good Models (ICML 2025 · similarity 0.86)
• Learning and Forgetting Unsafe Examples in Large Language Models (ICML 2024 · similarity 0.86)
• Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning Attack (ICML 2025 · similarity 0.86)