Research map / Large language models / Language, safety and interpretability

Beyond Prompt Engineering: A Reinforced Token-Level Input Refinement for Large Language Models

AAAI 2025 · Large language models › Language, safety and interpretability · approach: Controlled text generation · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RTLIR, a reinforcement-learning (Q-learning with dueling decomposition and prioritized replay) token-level input refinement module that decides keep/remove per token to improve LLM accuracy on PAWS-X and LM-Eval tasks.

Problem

Irrelevant information in LLM inputs wastes compute and degrades output quality, and prompt-engineering-based input refinement is context-dependent and biased by expert experience.

Approach

Formulates input refinement as an MDP where each token is a state with a binary keep/delete action, learned by a Q-learning agent using immediate (embedding cosine similarity) and terminal (log-probability improvement) rewards; the module is plug-and-play and LLM-agnostic.

Key results

• On PAWS-X, RTLIR improves average accuracy by 3.25% (Qwen2-1.5B), 2.51% (Gemma-2B), 11.03% (Llama-2-7B), 6.48% (Vicuna-7B-v1.3), and 6.51% (Llama-3-8B) over the original input.
• RTLIR improves Llama-2 on 14 of 16 LM-Eval tasks, e.g., WNLI 45.07% to 47.89% and CoLA MCC -2.33 to 0.58.
• Ablation shows removing terminal reward reduces average accuracy by about 10%; immediate-reward threshold of 0.05 performs best.

Measured effects reported in the paper

Surprising findings

• Refinement decreased accuracy on French PAWS-X for most models (e.g., -1.30 for Qwen2-1.5B), which the authors attribute to French language structure.

Limitations

• Evaluation limited to multiple-choice/classification-style tasks and five models; no generation benchmarks.
• Reported gains vary widely across languages and models; French results are negative for most models.
• Reward design relies on hand-set threshold beta and an undefined optimal refinement sequence v*.

Experimental setup

Datasets: PAWS-X Language Model Evaluation Harness tasks (16 datasets incl. CoLA, MRPC, QQP, WNLI, SOCIAL IQA, RACE, MMLU subjects, KorMe

Benchmarks: PAWS-X CoLA MRPC QQP WNLI SOCIAL IQA KorMedMCQA RACE MMLU subjects

Metrics: accuracy MCC F1 exact match

Baselines: Origin (unrefined input) prompt engineering

Builds on: Deep Q-learning (dueling network) prioritized experience replay Markov decision process

Closest papers by meaning

• Unleashing the Potential of Large Language Models as Prompt Optimizers: Analogical Analysis with Gradient-based Model Optimizers (AAAI 2025 · similarity 0.88)
• Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass (ICLR 2025 · similarity 0.88)
• GReaTer: Gradients Over Reasoning Makes Smaller Language Models Strong Prompt Optimizers (ICLR 2025 · similarity 0.88)
• Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting (ICLR 2024 · similarity 0.88)
• Prompt Sketching for Large Language Models (ICML 2024 · similarity 0.88)
• Scaling Prompt Synthesis for Large Language Model Reasoning (ICML 2026 · similarity 0.88)