Research map / Large language models / Agents, code and math
Do LLMs ``know'' internally when they follow instructions?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Probes LLM first-token representations to find an instruction-following direction that predicts compliance on unseen tasks (not unseen instruction types) and steers success rates upward versus random directions.
Problem
LLMs often fail to follow simple, clear instructions, and it is unclear whether their internal states encode whether an instruction will be followed.
Approach
Train linear probes on hidden representations (first, middle, last token) to predict instruction-following success on IFEval-simple, identify an instruction-following direction, use it for representation engineering to convert failures into successes, and analyze its relation to task familiarity, instruction difficulty, and phrasing.
Key results
- Linear probes on first-token early-layer representations reach AUROC ~0.77-0.88 for unseen tasks but only ~0.50-0.56 for unseen instruction types.
- Representation engineering along the instruction-following direction raises success rate over original outputs and random directions, e.g., LLaMA-2-13B SR 0.61 to 0.65 and Phi-3 0.71 to 0.74.
- Phrasing perturbations align more with the instruction-following direction than task familiarity or instruction difficulty perturbations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train a linear probe on first-token representations for a consistent instruction type and test on unseen tasks | LLaMA-2-chat-7B, LLaMA-2-chat-13B, Mistral-7B-Instruct-v0.3, and Phi-3-mini-128k · IFEval-simple, unseen tasks | AUROC: 0.77 $\pm$ 0.04; 0.83 $\pm$ 0.03; 0.74 $\pm$ 0.02; 0.88 $\pm$ 0.03 | Helped |
| Train a linear probe on four instruction types and test on an unseen instruction type | Four evaluated language models · IFEval-simple, unseen instruction types | AUROC: 0.52 $\pm$ 0.03; 0.56 $\pm$ 0.06; 0.50 $\pm$ 0.05; 0.55 $\pm$ 0.04 | No clear effect |
| Use the first or last token rather than a middle token representation | Linear probes for instruction-following · IFEval-simple, unseen tasks | AUROC: 0.77 $\pm$ 0.04 vs 0.55 $\pm$ 0.07; 0.73 $\pm$ 0.04 | Helped |
| Use early rather than later layers for task generalization | Linear probes for instruction-following · IFEval-simple, unseen tasks | AUROC: 0.83 $\pm$ 0.03; 0.81 $\pm$ 0.02; 0.78 $\pm$ 0.04 | Mixed |
| Shift representations along the instruction-following direction instead of leaving them unchanged | LLaMA-2-chat-7B · IFEval-simple | success rate: 0.57 $\pm$ 0.00 -> 0.59 $\pm$ 0.00 | Helped |
| Shift representations along the instruction-following direction instead of leaving them unchanged | LLaMA-2-chat-13B · IFEval-simple | success rate: 0.61 $\pm$ 0.00 -> 0.65 $\pm$ 0.02 | Helped |
| Shift representations along the instruction-following direction instead of leaving them unchanged | Mistral-7B-Instruct-v0.3 · IFEval-simple | success rate: 0.58 $\pm$ 0.00 -> 0.64 $\pm$ 0.02 | Helped |
| Shift representations along the instruction-following direction instead of leaving them unchanged | Phi-3-mini-128k · IFEval-simple | success rate: 0.71 $\pm$ 0.00 -> 0.74 $\pm$ 0.01 | Helped |
| Use the instruction-following direction rather than random directions | Four evaluated language models · IFEval-simple | success rate: 0.55 $\pm$ 0.00 -> 0.59 $\pm$ 0.00; 0.54 $\pm$ 0.12 -> 0.65 $\pm$ 0.02; 0.56 $\pm$ 0.02 -> 0.64 $\pm$ 0.02; 0.63 $\pm$ 0 | Helped |
| Shift representations along the instruction-following direction rather than leave them unchanged | Four evaluated language models · IFEval-simple | quality ratio: 0.87 $\pm$ 0.09 -> 0.87 $\pm$ 0.08; 0.92 $\pm$ 0.00 -> 0.94 $\pm$ 0.00; 0.95 $\pm$ 0.02 -> 0.98 $\pm$ 0.06; 0.76 $\pm$ 0 | No clear effect |
| Increase the training set to 23 instruction types (ablation) | Linear probes for instruction-following · IFEval-simple-expanded, unseen instruction types | AUROC | No clear effect |
| Shift representations opposite to the instruction-following direction (ablation) | Mistral-7B-Instruct-v0.3 and Phi-3-mini-128k · IFEval-simple | success rate: 0.58 ± 0.00 -> 0.54 ± 0.01; 0.71 ± 0.00 -> 0.60 ± 0.02 | Hurt |
Surprising findings
- Probes trained on first-token representations are as informative as last-token ones, suggesting success is predictable before response generation.
- Probes fail to generalize across instruction types even when expanded to 23 types, suggesting no single global instruction-following dimension.
Limitations
- Only four models and a narrow set of simple, verifiable instruction types were studied.
- Instruction-type generalization fails, limiting the claim of a global instruction-following dimension.
- Interpretation of the dimension as phrasing-related is based on a perturbation-based cosine similarity analysis and remains unclear.
Experimental setup
Datasets: IFEval IFEval-simple IFEval-simple-expanded
Benchmarks: IFEval-simple IFEval
Metrics: AUROC success rate quality ratio success conversion ratio success preservation ratio
Baselines: random directions original outputs
Builds on: IFEval representation engineering linear probing
Closest papers by meaning
- Do LLMs estimate uncertainty well in instruction-following? (ICLR 2025 · similarity 0.91)
- Generalizing Verifiable Instruction Following (NeurIPS 2025 · similarity 0.90)
- Can Large Language Models Understand Real-World Complex Instructions? (AAAI 2024 · similarity 0.89)
- Investigating the Effectiveness of Task-Agnostic Prefix Prompt for Instruction Following (AAAI 2024 · similarity 0.89)
- Do LLMs Build World Representations? Probing Through the Lens of State Abstraction (NeurIPS 2024 · similarity 0.88)
- Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.