Research map / Large language models / Retrieval and knowledge
Putting People in LLMs’ Shoes: Generating Better Answers via Question Rewriter
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a question rewriter with DPO on preference pairs built from automatic LFQA evaluation criteria, so that black-box LLMs produce better long-form answers without human feedback or multi-round refinement.
Problem
Vague or poorly formulated user questions hinder black-box LLMs from giving precise long-form answers; prior prompt optimization focused on task-level instructions or multi-round instance-level refinement with human/LLM feedback, mostly on arithmetic reasoning and short-form QA.
Approach
Single-round instance-level question rewriting: sample rewritten questions from an instruct LLM, label better/worse rewrites by whether generated answers dominate the original answer on automatic LFQA criteria, and optimize the rewriter with DPO (plus a validation-based preference score for checkpoint selection).
Key results
- On K-QA with GPT-4o, the rewriter raises S_comp from 0.5167 to 0.6253 while S_cont stays at 0.2400 (vs 0.2800 original).
- On Llama-3-8B-instruct (the training answer model), S_overall on TruthfulQA rises from 0.7397 to 0.7789 and S_pref on OASST1QA from 0.8654 to 0.9104.
- On K-QA with Llama-3-8B, the proposed method improves both S_comp and S_cont over the original, whereas PRoMPTed raises S_comp to 0.5012 but worsens S_cont to 0.5400.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune an instance-level question rewriter with DPO preference pairs selected using the Best-Random sampling combination | Llama-3-8B question answering · K-QA | S_comp: 0.4573 -> 0.4600 | Helped |
| Fine-tune an instance-level question rewriter with DPO preference pairs selected using the Best-Random sampling combination | Llama-3-8B question answering · K-QA | S_cont: 0.4400 -> 0.4000 | Helped |
| Fine-tune an instance-level question rewriter with DPO preference pairs | Llama-3-8B question answering · TruthfulQA | S_overall: 0.7397 -> 0.7789 | Helped |
| Fine-tune an instance-level question rewriter with DPO preference pairs | Llama-3-8B question answering · OASST1QA | S_pref: 0.8654 -> 0.9104 | Helped |
| Train the rewriter on K-QA rather than use the original questions | Llama-3-8B question answering · TruthfulQA | S_overall: 0.7397 -> 0.7454 | Helped |
| Train the rewriter on TruthfulQA rather than use the original questions | Llama-3-8B question answering · K-QA | S_comp: 0.4573 -> 0.4104 | Hurt |
| Train the rewriter on OASST1QA rather than use the original questions | Llama-3-8B question answering · TruthfulQA | S_overall: 0.7397 -> 0.7155 | Hurt |
| Use randomly sampled better questions and worst-ranked negative questions instead of the Best-Random combination (ablation) | DPO question rewriter · K-QA | S_comp: 0.4600 -> 0.4276 | Hurt |
| Use top-ranked better questions and worst-ranked negative questions instead of the Best-Random combination (ablation) | DPO question rewriter · K-QA | S_comp: 0.4600 -> 0.4026 | Hurt |
| Randomly sample both better and worse questions instead of the Best-Random combination (ablation) | DPO question rewriter · K-QA | S_comp: 0.4600 -> 0.4564 | No clear effect |
Surprising findings
- Rewriters trained on OASST1QA are almost ineffective on K-QA and TruthfulQA, while rewriters trained on K-QA and TruthfulQA transfer to OASST1QA.
- The Zero-Shot CoT task-level prompt often lowers S_overall on TruthfulQA, and the initial untuned rewriter frequently hurts answer quality.
Limitations
- Gains are mixed on some metrics; e.g., S_info drops for Rw-K on TruthfulQA and S_cont is not improved for every model.
- Automatic criteria use Llama-3-8B-instruct as judge for K-QA and a fine-tuned Llama-3-8B evaluator for TruthfulQA instead of the original GPT-based evaluators, with limited judge accuracy (about 70% on K-QA).
- Evaluated on only three LFQA datasets with small training sets (101 to 1,000 samples) and only comparisons on K-QA for other rewriting methods.
Experimental setup
Datasets: K-QA TruthfulQA OASST1QA
Benchmarks: K-QA TruthfulQA OASST1QA
Metrics: S_comp S_cont S_truth S_info S_overall S_pref
Baselines: Original question Zero-Shot CoT Initial Rewriter PRewrite (DPO replication) T5-Flan-base ill-formed question rewriter PRoMPTed
Builds on: direct preference optimization PRewrite FActScore TruthfulQA Zero-Shot CoT
Closest papers by meaning
- Rethinking Generative Large Language Model Evaluation for Semantic Comprehension (ICML 2024 · similarity 0.88)
- RetLLM-E: Retrieval-Prompt Strategy for Question-Answering on Student Discussion Forums (AAAI 2024 · similarity 0.88)
- RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content (NeurIPS 2024 · similarity 0.88)
- Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questions (ICLR 2025 · similarity 0.88)
- QGen Studio: An Adaptive Question-Answer Generation, Training and Evaluation Platform (AAAI 2025 · similarity 0.87)
- Cost-efficient Knowledge-based Question Answering with Large Language Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.