Research map / Large language models / Instruction and fine-tuning
Residual Stream Analysis of Overfitting And Structural Disruptions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
The paper links repetitive safety fine-tuning data to false refusals, introduces FlowLens (a PCA over concatenated multi-layer residuals) to analyze residual geometry, and proposes Variance Concentration Loss (VCL), an auxiliary regularizer on mid-layer residual variance, which cuts XSTest false refusals by over 35 points.
Problem
Safety fine-tuning on repetitive refusal-templated data causes false refusals (benign queries declined), and the internal representational cause is not well understood.
Approach
Analyze residual-stream geometry by concatenating residuals across all layers and applying a single PCA (FlowLens), observe that safety data concentrates variance in top components, and add a loss that penalizes this variance concentration in mid-layer residuals during SFT.
Key results
- VCL reduces XSTest false refusal to 0.86 on Llama-3.2-1B versus 0.51 for the SFT baseline (over 35 points), while keeping MMLU at 0.42 and GSM8K at 0.51.
- False refusal rate on XSTest rises from 63% to 84% as safety data share grows from 0% to 40% in the analysis experiment.
- Safety completions show lower token entropy H1 (about 9.18 vs 12.05 for control) and distinct 2-gram rate (0.048 vs 0.205).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Variance Concentration Loss | Llama-3.2-1B-SFT · XSTest | score: 0.78 -> 0.86 | Helped |
| add Variance Concentration Loss | Llama-3.1-8B-SFT · XSTest | score: 0.80 -> 0.89 | Helped |
| add Variance Concentration Loss | Llama-3.2-1B-SFT · JBB | score: 0.91 -> 0.86 | Hurt |
| add Variance Concentration Loss | Llama-3.1-8B-SFT · JBB | score: 0.93 -> 0.89 | Hurt |
| add Variance Concentration Loss | Llama-3.2-1B-SFT · MMLU | score: 0.45 -> 0.42 | Hurt |
| add Variance Concentration Loss | Llama-3.1-8B-SFT · GSM8K | score: 0.63 -> 0.58 | Hurt |
| increase safety data ratio from 0.0 to 0.4 | Llama-3.2-1B · XSTest | False Refusal (%): 0.63 -> 0.84 | Hurt |
| increase safety data ratio from 0.0 to 0.4 | Llama-3.2-1B · XSTest | Score: 1.0 -> 0.85 | Hurt |
| vary principal component cutoff k (ablation) | VCL · XSTest and structural metrics | false refusal rate | No clear effect |
| increase regularization weight γ too much at small k (ablation) | VCL · general capability tasks | helpfulness | Hurt |
| set γ to 1.0×50 (ablation) | VCL · safety and structural stability | overall performance | Helped |
| select residual window depth [0.3,0.5] (ablation) | VCL · safety and structural stability | trade-off | Helped |
Surprising findings
- Cosine similarity between final-token residuals is highly unstable under removal of a trailing question mark, while PCA-based projections remain stable.
- Adding safety data increases false refusals while the top-PC alignment score falls, suggesting a link between geometric collapse and over-refusal.
Limitations
- Analysis focuses only on the leading intrinsic-dimension principal components, possibly missing lower-variance structure.
- A fixed intrinsic dimension is applied across all layers and prompts.
- Evaluation is restricted to a few Llama-family models and a limited set of safety datasets.
Experimental setup
Datasets: WildJailbreak WildGuardMix Tulu-3-SFT-Mixture TruthfulQA XSTest
Benchmarks: DAN HarmBench ToxiGen JBB OKTest ORB-H XSTest MMLU GSM8K BBH CodexEval
Metrics: accuracy false refusal rate lexical entropy distinct n-gram rate alignment score
Baselines: Llama-3.2-1B-SFT System Prompting DRO Self-CD Vector Ablation
Builds on: Tulu 3 SFT recipe TwoNN intrinsic dimension Self-Contrastive Decoding DRO
Closest papers by meaning
- OR-Bench: An Over-Refusal Benchmark for Large Language Models (ICML 2025 · similarity 0.89)
- SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal (ICLR 2025 · similarity 0.88)
- Fail-Closed Alignment for Large Language Models (NeurIPS 2026 · similarity 0.88)
- Learning and Forgetting Unsafe Examples in Large Language Models (ICML 2024 · similarity 0.88)
- Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.87)
- SCANS: Mitigating the Exaggerated Safety for LLMs via Safety-Conscious Activation Steering (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.