Research map / Large language models / Language, safety and interpretability

The Illusion of Role Separation: Hidden Shortcuts in LLM Role Learning (and How to Fix Them)

ICML 2025 · Large language models › Language, safety and interpretability · approach: Task vectors and steering

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that LoRA-finetuned LLMs learn shortcuts (task-type association and proximity to begin-of-text) for role separation rather than true role distinction, and proposes Position-enhanced fine-tuning (PFT), which shifts position IDs between system and user segments to improve separation.

Problem

Fine-tuned LLMs that handle multiple input roles (system vs. user) may not truly learn role separation; instead they may memorize attack patterns or exploit spurious cues, so prompt-injection defense results may not reflect genuine role differentiation.

Approach

Use a controlled benign-train / adversarial-eval framework to expose shortcuts (task-type association, proximity to begin-of-text), and reinforce invariant role signals by manipulating position IDs: insert a fixed gap d between system and user sections while preserving within-section order.

Key results

• On Llama-3-8B, PFT raises TensorTrust Extraction accuracy from 33% (SFT) to 62% on dataset-initial, and Gandalf Ignore from 86% to 94%.
• On dataset-symm, PFT variants consistently outperform baselines across all attacks on Llama and Gemma models, per the paper's claim.
• PFT keeps password-task accuracy, Alpaca log-likelihood and KL divergence to the base model stable relative to standard SFT.

Measured effects reported in the paper

Surprising findings

• Fine-tuned models treat user-inserted override instructions almost identically whether placed in the system or user role (99.8% vs 99.2% refusal), while the base model distinguishes them (100% vs 74%).
• Models may treat tokens nearest begin-of-text as privileged system instructions, so inserting generic text before the key instruction causes large failures, while inserting after has smaller effects.

Limitations

• Experiments limited to two-role closed-domain setting with Llama-3-8B and Gemma-2-9b; generalization to other models and open-domain settings untested.
• The role of d (gap size) must be selected by validation loss; results depend on this hyperparameter.
• Some claims about shortcut mechanisms (e.g., attention sink explanation) are hypotheses rather than verified.

Experimental setup

Datasets: dataset-initial (GPT-4 generated ambiguous user inputs, Llama-3-8B responses) dataset-symm (symmetrized augmentation) Gandalf Summarization Gandalf Ignore TensorTrust extraction TensorTrust hijacking Alpaca

Benchmarks: Gandalf Summarization Gandalf Ignore TensorTrust Extraction TensorTrust Hijacking Password dataset Alpaca

Metrics: attack success / accuracy log-likelihood KL divergence

Baselines: Vanilla SFT Delimiter-enhanced SFT Data-augmented SFT

Builds on: LoRA instruction hierarchy position ID manipulation for long-context extension

Closest papers by meaning

• Context-Aware Hierarchical Learning: A Two-Step Paradigm towards Safer LLMs (NeurIPS 2025 · similarity 0.86)
• Enhancing Multiple Dimensions of Trustworthiness in LLMs via Sparse Activation Control (NeurIPS 2024 · similarity 0.85)
• The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (ICLR 2024 · similarity 0.85)
• Keeping an Eye on LLM Unlearning: The Hidden Risk and Remedy (NeurIPS 2025 · similarity 0.85)
• Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superposition (ICML 2025 · similarity 0.85)
• Concept Incongruence: An Exploration of Time and Death in Role Playing (NeurIPS 2025 · similarity 0.85)