Research map / Large language models / Language, safety and interpretability
Selective Prompt Anchoring for Code Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Selective Prompt Anchoring (SPA), a training-free logit-arithmetic method that amplifies the influence of user-prompt tokens by combining logits from original and mask-replaced prompts, to counter attention dilution in code generation.
Problem
LLMs pay progressively less attention to the user prompt as they generate more code tokens (attention dilution), which is linked to code generation errors.
Approach
Approximates the effect of scaling a prompt segment's semantic influence by a first-order (Taylor) approximation: augmented logits = w * f(original) + (1-w) * f(prompt with anchored tokens masked), requiring no weight updates and no attention-head editing.
Key results
- SPA improves Pass@1 by up to 12.9 absolute points (DeepSeek-Coder-6.7B on HumanEval, 75.6 to 88.5), with consistent gains across 6 models and 6 benchmarks.
- With SPA, DeepSeek-Coder-6.7B (88.5 on HumanEval) surpasses the base DeepSeek-Coder-33B (81.7).
- SPA yields a 5.9x larger Pass@1 gain than PASTA while using about 20% of its inference time, and outperforms prompting methods like Self-Debugging and ReAct.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply SPA to code generation | six code LLMs · HumanEval, HumanEval+, MBPP, MBPP+, BigCodeBench, LiveCodeBench | Pass@1: up to 12.9% | Helped |
| include source code in the anchored text (ablation) | SPA with natural-language anchoring · HumanEval | Pass@1 improvement (%): 5.48 -> 4.87 | Hurt |
| include test cases in the anchored text (ablation) | SPA with natural-language anchoring · HumanEval | Pass@1 improvement (%): 5.48 -> 5.11 | Hurt |
| include source code and test cases in the anchored text (ablation) | SPA with natural-language anchoring · HumanEval | Pass@1 improvement (%): 5.48 -> 4.76 | Hurt |
| anchor human-labeled important tokens instead of natural language (ablation) | SPA · HumanEval | Pass@1 improvement (%): 5.48 -> 3.66 | Hurt |
| apply SPA across prompt-length subsets | six code LLMs · HumanEval, short prompts | Pass@1 improvement (%): 1.1% average | Mixed |
| apply SPA across prompt-length subsets | six code LLMs · HumanEval, medium prompts | Pass@1 improvement (%): 3.2% average | Mixed |
| apply SPA across prompt-length subsets | six code LLMs · HumanEval, long prompts | Pass@1 improvement (%): 11.7% average | Helped |
| increase anchoring strength beyond its optimum (ablation) | SPA · code generation | Pass@1 | Hurt |
| use SPA for GSM8K | Llama-3.1-Instruct-8B · GSM8K | performance: + 1.10% | Helped |
| use SPA for MMLU | Llama-3.1-Instruct-8B · MMLU | performance: + 0% | No clear effect |
| use SPA for TruthfulQA Truthfulness | Llama-3.1-Instruct-8B · TruthfulQA (Truth.) | performance: + 3.14% | Helped |
Surprising findings
- Anchoring only the natural language description outperforms anchoring with source code or test cases, and human-labeled important tokens yield the smallest gains.
- On MMLU, SPA gives no improvement, which the authors attribute to short outputs causing little attention dilution.
Limitations
- Anchored text selection and anchoring strength are fixed and pre-defined, requiring per-model/dataset tuning via grid search.
- Relies on test cases to decide when to activate SPA, which may not be available in all settings.
- Gains on BigCodeBench and LiveCodeBench are modest in absolute terms, and SPA doubles memory or compute for the masked forward pass.
Experimental setup
Datasets: HumanEval MBPP HumanEval+ MBPP+ BigCodeBench LiveCodeBench
Benchmarks: HumanEval HumanEval+ MBPP MBPP+ BigCodeBench LiveCodeBench HumanEval-X TruthfulQA GSM8K MMLU BoolQ
Metrics: Pass@1 Pass@10 latency (tokens/sec)
Baselines: PASTA Self-Debugging Self-Planning ReAct Self-Edit Base model greedy decoding
Builds on: PASTA anchoring effect contrastive decoding
Closest papers by meaning
- What Makes Large Language Models Reason in (Multi-Turn) Code Generation? (ICLR 2025 · similarity 0.86)
- Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting (ICLR 2024 · similarity 0.85)
- SelfCodeAlign: Self-Alignment for Code Generation (NeurIPS 2024 · similarity 0.84)
- Online Prompt Selection for Program Synthesis (AAAI 2025 · similarity 0.84)
- Training Language Models to Generate Quality Code with Program Analysis Feedback (NeurIPS 2025 · similarity 0.84)
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.