Research map / Large language models / Alignment and preferences

Exploring Intrinsic Alignments Within Text Corpus

AAAI 2025 · Large language models › Alignment and preferences · approach: Steering and correcting LLM behavior · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ISAAC, a human-feedback-free safety alignment that clusters a dialogue corpus by context and response, then samples sharpened pseudo-responses from the long-tailed cluster distribution to fine-tune LLMs or train a T5 rephrasing model.

Problem

Safety alignment of LLMs and dialogue models normally needs costly human annotations (SFT responses or RLHF preferences) and can miss topics not covered by annotated subsets; the paper asks whether safety signal can be extracted from the raw text corpus itself.

Approach

Assumes safe responses dominate the corpus in a long-tailed distribution over response clusters within each context cluster; uses an adaptively sharpened, temperature-warped softmax over cluster sizes to sample pseudo-safe responses, trained with multi-target and tempering supervision, with a theorem arguing convex sampling raises the chance of safe responses.

Key results

• On LLM experiments, ISAAC improved safety of Claude, Llama-3 and Qwen2 by 1.4%, 9.5% and 9.9% over comprehensive alignment (as reported in the introduction).
• ISAAC rephrasing reduced task-level unsafe rate (DPR) by at least 69% on TOD models with only about 1% drop in success rate and BLEU-4.
• ISAAC increased diversity (0.03 DIST-2 and 0.62 Entropy gains) and lowered perplexity while keeping safety comparable to SFT baselines.

Measured effects reported in the paper

Surprising findings

• Random sampling of pseudo-responses does not improve safety, especially under high poisoning fractions.
• Softmax-based sampling (exp) sometimes increased unsafe outputs when the corpus violated the safe-majority assumption, where winner-take-all performed better.

Limitations

• Relies on the assumption that safe responses form the majority and unsafe ones lie in distinct tail clusters; fails when the corpus is highly poisoned.
• Coarse-grained alignment that may lose informativeness on task-oriented dialogue.
• Safety evaluation uses a RoBERTa classifier with about 80.88% accuracy plus human checks on flagged responses.

Experimental setup

Datasets: DiaSafety MultiWoz 2.1

Benchmarks: DiaSafety MultiWoz 2.1 (poisoned)

Metrics: safety score Entropy DIST-2 B-PPL F-PPL acceptability (COLA) engagingness (SST) success rate BLEU-4 DPR

Baselines: Detoxify Perspective API BBF BAD GPT-4 Claude Llama-3 8B Qwen2 7B Phi-3 SimpleTOD SOLOIST AuGPT

Builds on: BAD (baked-in safe training) DPO BARTScore

Closest papers by meaning

• Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment (ICLR 2024 · similarity 0.90)
• Aligner: Efficient Alignment by Learning to Correct (NeurIPS 2024 · similarity 0.90)
• Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysis (ICLR 2024 · similarity 0.89)
• The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (ICLR 2024 · similarity 0.89)
• Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples (ICML 2025 · similarity 0.89)
• On Evaluating LLM Alignment by Evaluating LLMs as Judges (NeurIPS 2025 · similarity 0.89)