Research map / Large language models / Language, safety and interpretability
Unveiling and Manipulating Prompt Influence in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Token Distribution Dynamics (TDD), which projects every input token's hidden states through the LM head to score prompt-token saliency for contrastive LLM generation, and applies it to zero-shot toxicity suppression and sentiment steering.
Problem
Input saliency for autoregressive LLMs: existing saliency methods (gradient, attention/vector, perturbation) misalign with next-token generation objectives or rely on linearity assumptions that break down in deep LLMs, so prompt influence on outputs is poorly explained.
Approach
Uses the LM head to project each input token's intermediate and final hidden states onto vocabulary distributions and measures saliency as the change in target-minus-alternative probability dynamics, in forward, backward, or bidirectional directions, requiring only forward passes and no hyperparameters.
Key results
- On BLiMP across GPT2, GPTJ, BLOOM, Pythia and LLaMA2, TDD-bidirectional/backward reach about 70-71% AOPC on GPT2/GPTJ/BLOOM/Pythia versus roughly 64% for the best baseline (Con-GI).
- In toxic suppression with GPT2, TDD lowers Perspective toxicity to 0.20 from 0.26 for the best prior saliency/controlled baseline (Con-GI, FFNControl) while keeping fluency near 23.1.
- In sentiment steering, TDD yields 0.87 negative and 0.78 positive sentiment percentages, above FFNControl's 0.82 and 0.68.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TDD-backward instead of Con-GI for saliency explanations | GPT2-large · BLiMP (11 datasets) | AOPC (%): 64.71 -> 70.46 | Helped |
| Use TDD-bidirectional instead of Con-GI for saliency explanations | GPTJ-6B · BLiMP (11 datasets) | AOPC (%): 64.89 -> 71.05 | Helped |
| Use TDD-bidirectional instead of Con-GI for saliency explanations | BLOOM-7B · BLiMP (11 datasets) | AOPC (%): 65.18 -> 70.22 | Helped |
| Use TDD-backward instead of Con-GI for saliency explanations | Pythia-6.9B · BLiMP (11 datasets) | AOPC (%): 64.59 -> 71.29 | Helped |
| Use TDD-bidirectional instead of Con-GI for saliency explanations | LLaMA2-7B · BLiMP (11 datasets) | AOPC (%): 59.73 -> 65.51 | Helped |
| Neutralize salient toxic prompt triggers using TDD-bidirectional | GPT2 · REALTOXICPROMPTS subset | Toxicity: 0.24 -> 0.20 | Helped |
| Replace the salient sentiment cue with a negative steering token using TDD-bidirectional | GPT2 · Neutral to negative (OWT) | Negative percent: 0.82 -> 0.87 | Helped |
| Replace the salient sentiment cue with a positive steering token using TDD-bidirectional | GPT2 · Neutral to positive (OWT) | Positive percent: 0.68 -> 0.78 | Helped |
| Use TDD-bidirectional for multiple target and alternative tokens | GPT2-large · SST2 | AOPC (%): 56.27 -> 63.30 | Helped |
| Use TDD-bidirectional for multiple target and alternative tokens | GPT2-large · AG's News | AOPC (%): 66.34 -> 69.48 | Helped |
| Remove alternative tokens from TDD-forward (ablation) | TDD-forward · BLiMP (11 datasets), GPT2-large | AOPC (%): 67.28 -> 66.55 | Hurt |
| Remove alternative tokens from TDD-backward (ablation) | TDD-backward · BLiMP (11 datasets), GPT2-large | AOPC (%): 70.46 -> 67.48 | Hurt |
Surprising findings
- Forward-only TDD without alternative tokens still outperforms contrastive gradient baselines, suggesting alternative tokens are not required for good saliency.
Limitations
- TDD-backward and TDD-bidirectional need one pass per input token, making them slower than TDD-forward.
- Neutralizing triggers can reduce linguistic fluency in toxicity suppression.
- Evaluation is mostly on BLiMP minimal pairs and small set of controlled generation tasks with GPT2.
Experimental setup
Datasets: BLiMP (11 subsets) REALTOXICPROMPTS (1,225-prompt subset) OWT neutral prompts (5000) SenticNet word lists AG's News SST2
Benchmarks: BLiMP REALTOXICPROMPTS OWT AG's News SST2
Metrics: AOPC Sufficiency Toxicity (Perspective API attributes) Fluency (GPT2-XL perplexity) Dist-1/2/3 positive/negative sentiment percentage
Baselines: Integrated Gradients Contrastive Gradient Norm Contrastive Gradient x Input Rollout WORDFILTER FFNControl Style Prompting Augmented Style Prompting
Builds on: contrastive explanations for LLMs logit lens integrated gradients AtMan
Closest papers by meaning
- Leveraging Attention to Effectively Compress Prompts for Long-Context LLMs (AAAI 2025 · similarity 0.90)
- Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting (ICLR 2024 · similarity 0.88)
- The Devil is in the Prompts: Retrieval-Augmented Prompt Optimization for Text-to-Video Generation (CVPR 2025 · similarity 0.87)
- Interpreting the Repeated Token Phenomenon in Large Language Models (ICML 2025 · similarity 0.87)
- Measuring Non-Adversarial Reproduction of Training Data in Large Language Models (ICLR 2025 · similarity 0.87)
- Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.