Research map / Large language models / Alignment and preferences

Enhancing Multiple Dimensions of Trustworthiness in LLMs via Sparse Activation Control

NeurIPS 2024 · Large language models › Alignment and preferences · approach: LLM-as-judge and feedback data · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Sparse Activation Control, a training-free method that uses path patching to find task-relevant attention heads and models their activations with a Gaussian (GMM-style) transform to steer Llama2-13B-Chat on adversarial factuality, preference bias, and exaggerated safety simultaneously.

Problem

Representation engineering controls for trustworthiness degrade when multiple behaviors are steered simultaneously, because concept directions from PCA over all layers/MLPs overlap and interfere across tasks.

Approach

Identify sparse, near-non-overlapping attention heads per task via path patching (causal mediation), model each task's head activations with a single Gaussian per class instead of PCA directions, and map activations from the non-target to the target distribution, enabling near-independent multi-task control.

Key results

• On Llama2-13B-Chat, SAC multi-task reaches 86.12% CR on Adv Factuality, 53.75% RR on Preference Bias and 88.5% NRR on Exaggerated Safety, versus 76.56%, 10.83%, 67% with no control.
• RepE-Mean and RepE-Merge in multi-task settings fall below no-control on several tasks (e.g., Pref Bias 5% and 10% RR), while SAC retains most single-task gains.
• Path patching finds heads with low cross-task overlap (most task pairs under 10%), and SAC generalizes to Qwen2-7B and Qwen2-72B-Instruct with multi-task Pref Bias RR of 58.33% and 80.83%.

Measured effects reported in the paper

Surprising findings

• Multi-task RepE control performs worse than no control on several tasks due to mixed PCA directions losing task-specific meaning.
• Qwen2 models refuse none of the preference bias questions by default (0% RR), showing over-alignment toward always choosing an option.

Limitations

• Only three trustworthiness dimensions (plus robustness and privacy in appendix) are validated; many other trust aspects remain unexplored.
• Adv factuality evaluation via GPT-4 can be unreliable when correcting misinformation is beyond the model's ability.
• Uses a single Gaussian per task instead of a full mixture, a simplification of the linear representation assumption.

Experimental setup

Datasets: golden_advfactuality PreferenceBias XSTEST AdvBench MMLU CSQA robustness dataset from TrustLLM privacy dataset from TrustLLM

Benchmarks: golden_advfactuality PreferenceBias XSTEST MMLU CSQA AdvBench

Metrics: Correct Rate Refusal Rate Not Refusal Rate accuracy refuse-to-answer rate

Baselines: No Control RepE RepE-Mean RepE-Merge RepE-CLS

Builds on: Representation Engineering (RepE) path patching causal mediation analysis TrustLLM

Closest papers by meaning

• Position: TrustLLM: Trustworthiness in Large Language Models (ICML 2024 · similarity 0.89)
• More RLHF, More Trust? On The Impact of Preference Alignment On Trustworthiness (ICLR 2025 · similarity 0.89)
• Trustworthy Alignment of Retrieval-Augmented Large Language Models via Reinforcement Learning (ICML 2024 · similarity 0.88)
• Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.88)
• Sparsity-Guided Holistic Explanation for LLMs with Interpretable Inference-Time Intervention (AAAI 2024 · similarity 0.88)
• Interpreting Learned Feedback Patterns in Large Language Models (NeurIPS 2024 · similarity 0.87)