Research map / Federated learning and privacy / Differential privacy

Differentially Private Steering for Large Language Model Alignment

ICLR 2025 · Federated learning and privacy › Differential privacy · approach: Differentially private synthetic data · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PSA, a differentially private activation-editing (steering) method that adds calibrated Gaussian noise to clipped mean steering vectors computed from private demonstrations, and introduces a canary-based membership inference attack for steering.

Problem

Activation-editing steering of LLMs built from private demonstration datasets can leak membership or content of those private samples, and no formal privacy guarantees existed for this setting.

Approach

Compute per-layer mean difference steering vectors from positive/negative demonstrations after L2 clipping and scaling, then add Gaussian noise so each layer's vector is (eps, delta)-DP; post-processing gives unlimited queries at fixed privacy cost, and a canary-based MIA audits empirical privacy.

Key results

• PSA achieves alignment accuracy comparable to non-private Mean Steering across seven behaviors and outperforms zero-shot for Llama, Mistral and Qwen, with theoretical epsilon around 2 for most datasets.
• On Llama-2-7B, the membership inference attack gives empirical epsilon 0.6 for PSA versus 4.0 for non-private Mean Steer on the hallucination dataset (theoretical epsilon 2.0).
• MMLU performance under PSA stays close to Mean Steer (e.g., Sycophancy 63.0 vs 64.0 for Llama-2 7B), indicating limited utility loss.

Measured effects reported in the paper

Surprising findings

• PSA sometimes outperforms non-private Mean Steering on some behaviors (e.g., Refusal and Corrigibility), which the authors attribute to DP noise not significantly changing the activation direction.
• Empirical epsilon values from the MIA are smaller than the theoretical guarantees, suggesting the bounds are conservative.

Limitations

• Utility degrades for smaller models (text generation quality limited at 0.5B-1.5B sizes).
• Utility decreases with larger noise scales and clipping thresholds, requiring hyperparameter tuning.
• Privacy composition is basic, so guarantee grows linearly with number of steered layers.

Experimental setup

Datasets: Anthropic Advanced AI Risk evaluation datasets (sycophancy, hallucination, refusal, survival instinct, myopic reward, AI MMLU

Benchmarks: Anthropic Advanced AI Risk behavioral multiple-choice datasets MMLU

Metrics: accuracy on behavioral multiple choice GPT-4 judged score (0-10) for open-ended generation MMLU average probability of correct answer empirical epsilon from MIA FPR/FNR

Baselines: Mean Steering PCA Steering zero-shot

Builds on: mean activation steering in-context vectors (PCA steering) differential privacy Gaussian mechanism DP-SGD

Closest papers by meaning

• Privacy-Preserving Instructions for Aligning Large Language Models (ICML 2024 · similarity 0.91)
• Aligning Large Language Models with Representation Editing: A Control Perspective (NeurIPS 2024 · similarity 0.89)
• PaCE: Parsimonious Concept Engineering for Large Language Models (NeurIPS 2024 · similarity 0.88)
• Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing (ICLR 2025 · similarity 0.88)
• Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack (NeurIPS 2024 · similarity 0.88)
• Spectral Editing of Activations for Large Language Model Alignment (NeurIPS 2024 · similarity 0.88)