Research map / Large language models / Language, safety and interpretability

When Bad Data Leads to Good Models

ICML 2025 · Large language models › Language, safety and interpretability · approach: Pruning and circuits in LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that pretraining Olmo-1B on a small fraction (~10%) of toxic 4chan data makes toxicity more linearly separable and easier to suppress with post-training (ITI, prompting, SFT, DPO), yielding lower output toxicity than clean-data baselines.

Problem

Pretraining data filtering removes toxic content, but post-training detoxification is known to bypass rather than remove toxic mechanisms; the question is whether including toxic pretraining data improves the alignability of the resulting model.

Approach

Treat pretraining and post-training as a co-designed system: adding a controlled proportion of toxic data yields a less entangled, more linearly separable toxicity representation, so activation steering (ITI) and other post-training methods can reduce generational toxicity more effectively than on clean-trained models, with a sweet spot around 10% toxic data.

Key results

• Olmo-1B trained with 10% 4chan data plus strong ITI reduces ToxiGen toxicity from 41.40 (clean) to 2.63 and RealToxicityPrompts from 31.15 to 7.11, with CE loss rising to 3.23.
• Under weak ITI, the 10%-toxic model reaches 16.25 ToxiGen toxicity with CE loss 2.65, outperforming MEDA (22.02, CE 2.71) and INST (18.99, CE 2.73) on the trade-off.
• Probe accuracy for toxicity is significantly higher with 25% 4chan (p = 0.0002), and under GCG red-teaming strong steering lowers attack success from 46% (clean) to 38.5% (10% toxic).

Measured effects reported in the paper

Surprising findings

• Toxic pretraining data increases base-model toxicity but makes it easier to remove via steering, producing a smile-shaped (non-monotonic) detoxification curve with a trough at about 10% toxic data.
• Unsteered toxicity rises with toxic data, yet steered toxicity falls up to 10% before rising again.

Limitations

• Sweet spot of ~10% toxic data is identified only for the small 1B scale and specific data; generalization to larger models is untested.
• Steering with strong intervention increases cross-entropy loss (alignment tax rises to 3.23 at 10% toxic data).

Experimental setup

Datasets: C4 4chan ToxiGen RealToxicityPrompts Jigsaw OpenWebText MMLU AdvBench Tulu V2 OpenHermes-2.5 WebInstructSub Code-Feedback

Benchmarks: ToxiGen RealToxicityPrompts MMLU AdvBench

Metrics: Perspective API toxicity score cross-entropy loss on OpenWebText MMLU accuracy GCG attack success rate probe validation accuracy

Baselines: prompting MEDA INST supervised finetuning DPO

Builds on: inference-time intervention superposition hypothesis direct preference optimization Welch bound

Closest papers by meaning

• Learning and Forgetting Unsafe Examples in Large Language Models (ICML 2024 · similarity 0.89)
• IF-Guide: Influence Function-Guided Detoxification of LLMs (NeurIPS 2025 · similarity 0.89)
• Redefining Experts: Interpretable Decomposition of Language Models for Toxicity Mitigation (NeurIPS 2025 · similarity 0.88)
• Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples (NeurIPS 2026 · similarity 0.88)
• PoisonBench: Assessing Language Model Vulnerability to Poisoned Preference Data (ICML 2025 · similarity 0.88)
• SafetyPrompts: A Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety (AAAI 2025 · similarity 0.88)