Research map / Large language models / Alignment and preferences

Personality Alignment of Large Language Models

ICLR 2025 · Large language models › Alignment and preferences · approach: Steering and correcting LLM behavior · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces personality alignment and the PAPI dataset of ~325k real questionnaire subjects, plus PAS, a training-free activation-steering method that tunes per-head directions and intervention strength to align Llama-3 models with individual personality profiles.

Problem

Standard LLM alignment targets general human values and fails to capture individual users' personality preferences; existing personalization is data-hungry, costly, or unreliable via prompting.

Approach

Probe attention heads for personality-related directions from questionnaire statements, select top-K heads, and search an intervention strength alpha per individual, adding offsets to head outputs without changing weights, so alignment requires only a small set of stored parameters and little optimization time.

Key results

• On PAPI with Llama-3-8B-Instruct, PAS reaches an overall Score of 8.89 versus 9.66 for Personalized-Soups, 10.88 for Prompt-MORL, 12.97 for DPO and 13.27 for PPO (lower is better).
• With Llama-3-70B-Instruct, PAS scores 9.21 versus 9.94 for the best other method (Few-Shot); GPT-4o few-shot scores 6.81, better than PAS on Dark Triad traits.
• PAS is reported to need about 1/5 to 1/6 of the optimization time of DPO/PPO, per the abstract and efficiency figure.

Measured effects reported in the paper

Surprising findings

• The larger Llama-3-70B-Instruct does not improve over the 8B model under PAS, which the authors attribute to broad knowledge hindering individual-specific alignment.
• GPT-4o few-shot prompting outperforms all white-box methods including PAS on the overall Score (6.81), and approximately 20% of GPT-4o queries were refused and excluded.

Limitations

• PAPI uses multiple-choice questionnaire responses with a demographically skewed annotator pool (mostly female, median age 25), limiting representativeness.
• Human evaluation uses only three to five evaluators, with self-reported IAA, and the Ethics Statement notes risks of echo chambers and manipulation.
• Dark Triad alignment remains weaker than GPT-4o, especially for Narcissism and Psychopathy in the smaller model.

Experimental setup

Datasets: PAPI (IPIP-NEO-120, IPIP-NEO-300, Short Dark Triad) LIMA GSM8K CommonSenseQA AddSub MultiArith SVAMP BigBench-Date StrategyQA Coin Flip

Benchmarks: PAPI GSM8K CommonSenseQA AddSub MultiArith SVAMP BigBench-Date StrategyQA Coin Flip LIMA

Metrics: Aligned Score (absolute deviation from individual's question accuracy win rate

Baselines: Few-Shot prompting Personality Prompt (P^2) PPO DPO Prompt-MORL Personalized-Soups GPT-4o

Builds on: Inference-Time Intervention DPO PPO Personality Prompt Personalized Soups

Closest papers by meaning

• Aligning to Thousands of Preferences via System Message Generalization (NeurIPS 2024 · similarity 0.89)
• Understanding the Learning Dynamics of Alignment with Human Feedback (ICML 2024 · similarity 0.88)
• Neuron based Personality Trait Induction in Large Language Models (ICLR 2025 · similarity 0.87)
• Open Models, Closed Minds? On Agents Capabilities in Mimicking Human Personalities through Open Large Language Models (AAAI 2025 · similarity 0.87)
• HPS: Hard Preference Sampling for Human Preference Alignment (ICML 2025 · similarity 0.87)
• Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.87)