Research map / Large language models / Language, safety and interpretability
Query-Dependent Prompt Evaluation and Optimization with Offline Inverse RL
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Prompt-OIRL, which learns an offline XGBoost reward model from existing prompt-benchmarking logs to evaluate query-prompt pairs without LLM calls, then selects prompts via best-of-N for query-dependent zero-shot arithmetic reasoning.
Problem
Zero-shot prompt optimization for arithmetic reasoning is usually query-agnostic, and evaluating prompts at inference without gold answers or repeated LLM calls is hard and expensive.
Approach
Treat prompt evaluation as offline inverse RL: reuse offline logs of prompt correctness collected when benchmarking existing prompts to train a proxy reward model on query and prompt embeddings, then pick the prompt with the highest proxy reward via best-of-N, without querying the LLM.
Key results
- Prompt-OIRL outperforms best-of-training selection by +24.3% in the scarce setting (K=1) and LMSC-style confidence baseline by +24.3% and +14.7% in the two settings.
- Learned reward model reaches about 0.96 accuracy on held-out MAWPS queries with GPT-3.5-turbo (Ours (Q) K=1 vs LMSC (Q) 0.662).
- Inference cost per query for Prompt-OIRL stays roughly constant (about 0.0004 USD with GPT-3.5) while LMSC grows to about 0.10 USD at K=110 prompts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use query-dependent Prompt-OIRL selection instead of the best-of-training strategy | Arithmetic reasoning prompt selection · Arithmetic reasoning tasks, averaged across tasks and LLMs | success rate: +24.3% | Helped |
| Use query-dependent Prompt-OIRL instead of a query-agnostic objective | Arithmetic reasoning prompt selection · Rich demonstration setting | success rate: +8.8% | Helped |
| Select from held-out prompts using the learned reward model | Prompt-OIRL · Rich demonstration setting | success rate: +1.8% | Helped |
| Use Prompt-OIRL instead of LLM confidence-based selection | Arithmetic reasoning prompt selection · Scarce demonstration setting | success rate: +24.3% | Helped |
| Use Prompt-OIRL instead of LLM confidence-based selection | Arithmetic reasoning prompt selection · Rich demonstration setting | success rate: +14.7% | Helped |
| Train the reward model with more prompt demonstrations (K=5 rather than K=1) (ablation) | Prompt-OIRL reward model · MAWPS and SVAMP held-out prompts | accuracy | Helped |
| Train the reward model with more prompt demonstrations (K=5 rather than K=1) (ablation) | Prompt-OIRL reward model · GSM8K held-out queries, LLaMA2-7B-Chat | accuracy: 0.773 -> 0.726 | Hurt |
| Use the learned reward model instead of LLM self-criticism | Prompt evaluation · Held-out queries across MAWPS, SVAMP, and GSM8K | accuracy and precision | Mixed |
| Use offline reward-model evaluation rather than LLM self-criticism | Prompt evaluation at inference · Single query with 110 prompt choices | inference cost: 0.35723 -> 0.00134 GPU hour; 0.09029 -> 0.00056 USD; 0.10237 -> 0.00062 USD | Helped |
Surprising findings
- Not using a prompt can sometimes outperform generally effective prompts, motivating query-dependent selection.
- Gradient boosting reward models were empirically much better than neural networks for this reward modeling task.
Limitations
- Reward model relies on offline logs from existing prompts and a fixed set of LLMs; generalization to unseen LLMs is not established.
- Best-of-N selection is a minimalist optimizer; more sophisticated policy optimization is left unexplored.
- Evaluated only on three arithmetic reasoning datasets.
Experimental setup
Datasets: GSM8K SVAMP MAWPS MultiArith
Benchmarks: GSM8K SVAMP MAWPS
Metrics: accuracy precision success rate inference cost (USD, GPU hours)
Baselines: LMSC (language models for self-criticism) LLM Confidence best-of-training prompt
Builds on: chain-of-thought prompting automatic prompt engineer RLHF
Closest papers by meaning
- GReaTer: Gradients Over Reasoning Makes Smaller Language Models Strong Prompt Optimizers (ICLR 2025 · similarity 0.89)
- Scaling Prompt Synthesis for Large Language Model Reasoning (ICML 2026 · similarity 0.89)
- Large Language Models as Optimizers (ICLR 2024 · similarity 0.88)
- QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentation (ICLR 2026 · similarity 0.88)
- Reinforcing General Reasoning Without Verifiers (ICLR 2026 · similarity 0.88)
- Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.