Research map / Large language models / Alignment and preferences
Measuring memorization in RLHF for code completion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measures how training data memorization propagates through the RLHF pipeline (fine-tuning, reward model training, RL fine-tuning) and compares it with direct preference learning via IPO, for code completion models.
Problem
Memorization of training data has been studied for pre-training and fine-tuning, but not for RLHF; it is unclear whether sensitive data used in reward modelling or RL fine-tuning of code completion models can be regurgitated.
Approach
Uses a counterfactual (control-model-based) definition of approximate memorization with sliding-window normalized edit distance, applied to synthetic crypto-code data with fictitious PII file paths, to track memorization across each RLHF stage and contrast it with IPO direct preference learning.
Key results
- Fine-tuned model memorized 319 of 3,526 SD.Base examples; 43-47% remain memorized after RL fine-tuning across KL settings.
- Only 0.9% of reward model training examples (SD.Base) are memorized by the RL fine-tuned model, versus 17.6% when directly fine-tuned on them.
- IPO on Gemma 2B and 7B memorizes 18.2% and 19.5% of SD.Base, far above RLFT's 0.9%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SD in reward model training instead of directly fine-tuning on it | Gemini Nano-1 (1.8B) RLHF · SD.Base | memorization rate: 17.6% -> 0.9% | Hurt |
| Use SD in reward model training instead of directly fine-tuning on it | Gemini Nano-1 (1.8B) RLHF · SD.Links | PII-like file path memorization rate: 50% -> 0% | Hurt |
| Lower the KL coefficient from α=0.5 to α=0.005 (ablation) | RLFT.1 initialized from FT.1 · SD.Links | PII-like file path memorization rate: 54.8% -> 12.6% | Hurt |
| Lower the KL coefficient from α=0.5 to α=0.1 (ablation) | RLFT.1 initialized from FT.1 · SD.Links | PII-like file path memorization rate: 54.8% -> 46.8% | Hurt |
| Lower the KL coefficient during RL fine-tuning (ablation) | RLFT.1 initialized from FT.1 · SD.Base | memorized examples remaining after RL fine-tuning: 43-47% | Mixed |
| Include preference data in IPO training rather than reward model training for RLHF | Gemma 2B · SD.Base | memorization rate: 0.9% -> 18.2% | Helped |
| Include preference data in IPO training rather than reward model training for RLHF | Gemma 7B · SD.Base | memorization rate: 0.9% -> 19.5% | Helped |
| Scale IPO from Gemma 2B to Gemma 7B | IPO · SD.Base | memorization rate: 18.2% -> 19.5% | Helped |
| Increase the RL fine-tuning learning rate from 3e-6 to 3e-5 (ablation) | RLFT.4 with α=0.005 · RL fine-tuning prompts | prompt memorization rate | Helped |
| Use IPO instead of RLHF with reward model training | Gemma 2B and Gemma 7B · SD.Links | PII-like file path memorization rate: 0% -> 0.6% | Helped |
| Scale up the RLFT model | RLHF · SD, CodeXGLUE, and LIMA | RL prompt memorization rate: <1% | Helped |
Surprising findings
- With a small KL penalty, RL fine-tuning collapsed to always outputting the string '#' because comments dominated positively labelled targets.
- Reward-model training data was memorized at very low rates (0% on SD.Links) even though the fine-tuned model regurgitated PII-like paths in 50% of completions.
Limitations
- Relies on a synthetic dataset with a fictitious library and fictitious PII paths, which may not reflect real user data.
- Uses a single fixed control model (FT.2) as an approximation of per-example counterfactual memorization.
- Memorization rates depend on KL penalty and learning rate hyperparameters, so conclusions may not transfer to other setups.
Experimental setup
Datasets: SD (synthetic, Gemini Ultra generated) PD (scraped permissive Python) CodeXGLUE CodeCompletion-line LIMA Anthropic HH
Benchmarks: HumanEval Infilling HumanEval SD CodeXGLUE CodeCompletion-line LIMA Anthropic HH
Metrics: counterfactual memorization rate (0.1-approximate, normalize HumanEval pass@1 HumanEval Infilling single/multi-line scores
Baselines: DPO CodeGemma 2B DeepSeek Coder 2B StarCoder2 3B Gemma 2B Gemma 7B
Builds on: RLHF IPO Fill-in-the-middle Gemma CodeGemma
Closest papers by meaning
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.90)
- How to Evaluate Reward Models for RLHF (ICLR 2025 · similarity 0.90)
- Provably Efficient Online RLHF with One-Pass Reward Modeling (NeurIPS 2025 · similarity 0.89)
- Avoiding exp(R) scaling in RLHF through Preference-based Exploration (NeurIPS 2025 · similarity 0.89)
- Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback (AAAI 2025 · similarity 0.89)
- Online Preference Alignment for Language Models via Count-based Exploration (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.