Research map / Large language models / Instruction and fine-tuning
Just Enough Shifts: Mitigating Over-Refusal in Aligned Language Models with Targeted Representation Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ACTOR, which fine-tunes a single transformer layer with a projection-calibrated refusal-direction loss on activations (no responses needed) to reduce over-refusal in aligned chat models while keeping safety.
Problem
Safety-aligned LLMs over-refuse benign but superficially harmful ("pseudo-harmful") prompts, hurting utility; existing training methods are data/compute heavy and inference-time steering methods are brittle and apply uniform shifts.
Approach
Identify a target layer via silhouette scores and a difference-in-means refusal direction, then fine-tune that layer so each query's activation is shifted by an amount proportional to its projection onto the refusal direction ("just enough" shift), using a cosine loss on activations rather than response supervision.
Key results
- ACTOR raises average compliance rate on Llama-2-7b-chat from 61.53% to 90.74% while AdvBench safety score stays at 99.03%.
- On Gemma-7b-it, ACTOR reaches average compliance 77.01% vs 71.76% default, with trade-off score 84.75 vs 82.88.
- ACTOR changes MMLU and MT-Bench by at most about 1.5% and keeps WikiText2 perplexity similar or lower.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune with ACTOR rather than use the default model | Llama-2-7b-chat · False-refusal benchmarks | Avg. C.R: 61.53 -> 90.74 | Helped |
| fine-tune with ACTOR rather than use the default model | Llama-2-7b-chat · AdvBench | Safety Score: 99.62 -> 99.03 | No clear effect |
| fine-tune with ACTOR rather than use the default model | Gemma-7b-it · False-refusal benchmarks | Avg. C.R: 71.76 -> 77.01 | Helped |
| fine-tune with ACTOR rather than use the default model | Llama-13b-chat · False-refusal benchmarks | Avg. C.R: 60.47 -> 84.10 | Helped |
| fine-tune with ACTOR rather than use the default model | Llama-2-7b-chat · MMLU | Avg. accuracy: 46.81 -> 47.05 | Helped |
| fine-tune with ACTOR rather than use the default model | Gemma-7b-it · MT-Bench | Avg.: 6.43 -> 6.33 | Hurt |
| fine-tune with ACTOR rather than use the default model | Llama-13b-chat · MT-Bench | Avg.: 6.67 -> 6.67 | No clear effect |
| use the silhouette-selected target layer 13 rather than layer 5 (ablation) | ACTOR on Llama-2-7b-chat · False-refusal benchmarks | Avg. C.R: 61.23 -> 90.74 | Helped |
| use the silhouette-selected target layer 13 rather than layer 25 (ablation) | ACTOR on Llama-2-7b-chat · False-refusal benchmarks | Avg. C.R: 61.62 -> 90.74 | Helped |
| use n=15 instead of n=50 benign calibration examples (ablation) | ACTOR · False-refusal benchmarks | Trade-off Score: 94.31 -> 94.17 | No clear effect |
| use n=10 instead of n=50 over-refusal prompts (ablation) | ACTOR · False-refusal benchmarks | Trade-off Score: 94.65 -> 94.32 | No clear effect |
| replace projection-calibrated individualized shifts with uniform shifts (ablation) | ACTOR · Safe, pseudo-harmful, and harmful queries | response coherence | Hurt |
Surprising findings
- Uniform refusal-direction shifts (as in inference-time steering) break the model, producing nonsensical outputs when applied in fine-tuning.
- Existing single-vector methods such as SCANS and Surgical are sensitive to which harmful dataset defines the refusal direction.
Limitations
- Over-refusal labels from public benchmarks reflect contested normative choices of their designers.
- Requires white-box access to model activations and an existing safety alignment to compute the refusal direction.
- Evaluated only on single-turn scenarios; multi-turn over-refusal is not measured.
Experimental setup
Datasets: HexPhi UltraChat XSTest SCOPE OR-Bench-Hard-1K PHTest TruthfulQA AdvBench OKTest MMLU MT-Bench WikiText2
Benchmarks: XSTest SCOPE OR-Bench-Hard PHTest OKTest AdvBench MMLU MT-Bench WikiText2
Metrics: compliance rate safety score trade-off score MMLU accuracy MT-Bench score perplexity
Baselines: SFT Safe-Decoding DRO Self-CD SCANS Surgical
Builds on: SCANS difference-in-means refusal direction representation fine-tuning DRO
Closest papers by meaning
- Latent-space Attacks for Refusal Evasion in Language Models (NeurIPS 2026 · similarity 0.91)
- OR-Bench: An Over-Refusal Benchmark for Large Language Models (ICML 2025 · similarity 0.90)
- EVOREFUSE: Evolutionary Prompt Optimization for Evaluation and Mitigation of LLM Over-Refusal to Pseudo-Malicious Instructions (NeurIPS 2025 · similarity 0.90)
- Fail-Closed Alignment for Large Language Models (NeurIPS 2026 · similarity 0.90)
- Refusal Direction is Universal Across Safety-Aligned Languages (NeurIPS 2025 · similarity 0.90)
- Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablation (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.