Research map / Large language models / Language, safety and interpretability
Extracting alignment data in open models
Federico Barbero, Xiangming Gu, Christopher A. Choquette-Choo, Chawin Sitawarin, Matthew Jagielski, Itay Yona, Petar Veličković, Ilia Shumailov, Jamie Hayes
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| prefill with full chat template instead of only the BOS token (ablation) | OLMo 2 13B extraction · OLMo 2 SFT set | mean embedding score: 0.857 -> 0.892 | Improved |
| measure memorisation with neural embeddings instead of string matching | extraction evaluation · OLMo 2 13B generations | memorisation rate | Improved |
| SFT on extracted synthetic data instead of original data | OLMo 2 7B · GSM8K | score: 0.7377 -> 0.7847 | Improved |
| SFT on extracted synthetic data instead of original data | OLMo 2 7B · BBH | score: 0.4944 -> 0.5161 | Improved |
| SFT on extracted synthetic data instead of original data | OLMo 2 7B · TQA | score: 0.5110 -> 0.5529 | Improved |
| SFT on extracted synthetic data instead of original data | OLMo 2 7B · IFE | score: 0.6617 -> 0.5028 | Hurt |
| SFT on extracted synthetic data instead of original data | OLMo 2 7B · MATH | score: 0.2077 -> 0.1705 | Hurt |
| SFT on extracted synthetic data instead of original data | OLMo 2 7B · MMLU | score: 0.6123 -> 0.6052 | No clear effect |
| RL on extracted synthetic dataset instead of ORZ 57k dataset | Qwen2.5 7B Dr. GRPO · AMC, MATH500, Minerva, OlympiadBench | accuracy | No clear effect |
| RL post-training (Open-Reasoner-Zero) raises likelihood of training prompts | Qwen2.5 7B · ORZ train prompts (142,770) | count above likelihood threshold: 18 -> 287 | Improved |
Closest papers by meaning
- Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples (ICML 2025 · similarity 0.87)
- Analyzing Similarity Metrics for Data Selection for Language Model Pretraining (NeurIPS 2025 · similarity 0.87)
- Scalable Extraction of Training Data from Aligned, Production Language Models (ICLR 2025 · similarity 0.87)
- Leveraging Importance Sampling to Detach Alignment Modules from Large Language Models (NeurIPS 2025 · similarity 0.87)
- Aligner: Efficient Alignment by Learning to Correct (NeurIPS 2024 · similarity 0.86)
- Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
