Research map / Large language models / Retrieval and knowledge
IQA-EVAL: Automatic Evaluation of Human-Model Interactive Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IQA-Eval, an LLM-agent framework that simulates users (with personas) to interact with IQA models and then scores the interactions, correlating with human judgments and benchmarking recent LLMs on HotpotQA and AmbigQA.
Problem
Evaluating LLMs for question answering in interactive, multi-turn human-model settings is expensive because it requires human participants; single-turn non-interactive metrics miss interaction quality such as helpfulness and fluency.
Approach
LLM-based Evaluation Agents (LEAs) simulate human users to generate multi-turn interactions with IQA models and then rate those interactions on helpfulness, fluency, queries and accuracy; assigning personas to LEAs better matches groups of human raters.
Key results
- IQA-Eval with GPT-4 as backbone reaches Pearson 0.652 on helpfulness, 0.591 on fluency and 0.613 overall against human judgments on the MMLU interaction data.
- Assigning the Expert persona raises the Expert-persona correlation to 0.756 (helpfulness) and 0.787 (fluency), with overall 0.670 to 0.690 range across personas.
- Benchmarking on 1000 HotpotQA and AmbigQA questions gives GPT-4 the highest accuracy (0.66 and 0.72) and the authors estimate human evaluation would cost $5k.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| assign personas to the LEA rather than use the standard persona (ablation) | IQA-Eval with GPT-3.5-turbo-1106 · MMLU interaction meta-evaluation | Pearson correlation (Overall): 0.510 -> 0.670; 0.634; 0.650; 0.690 | Helped |
| assign Expert persona (ablation) | IQA-Eval with GPT-3.5-turbo-1106 · MMLU interaction meta-evaluation | Pearson correlation (Fluency): 0.523 -> 0.787 | Helped |
| use multi-perspective evaluation by multiple LEAs (ablation) | IQA-Eval · MMLU interaction meta-evaluation | Pearson correlation (Overall): 0.613 -> 0.624; 0.551 -> 0.602; 0.510 -> 0.533 | Mixed |
| add impartiality and debiasing instructions to prompts (ablation) | IQA-Eval-GPT4 · MMLU interaction meta-evaluation | accuracy: 0.87 -> 0.87 | No clear effect |
| use an incorrect 20% Expert persona distribution instead of the Expert assignment (ablation) | IQA-Eval · MMLU interaction meta-evaluation | Pearson correlation (Helpfulness): 0.756 -> 0.708 | Hurt |
| use an incorrect 80% Expert persona distribution instead of the Expert assignment (ablation) | IQA-Eval · MMLU interaction meta-evaluation | Pearson correlation (Helpfulness): 0.756 -> 0.680 | Hurt |
| use Clarity-Seeker persona instead of the standard persona (ablation) | IQA-Eval with GPT-3.5-turbo-1106 · MMLU interaction meta-evaluation | Pearson correlation (Overall): 0.510 -> 0.690 | Helped |
| use Expert persona instead of the standard persona (ablation) | IQA-Eval with GPT-3.5-turbo-1106 · Text-Davinci accuracy | accuracy: 0.63 -> 0.73 | Helped |
| use Clarity-Seeker persona instead of the standard persona (ablation) | IQA-Eval with GPT-3.5-turbo-1106 · Text-Babbage accuracy | accuracy: 0.47 -> 0.57 | Helped |
| use Clarity-Seeker persona instead of the standard persona (ablation) | IQA-Eval with GPT-3.5-turbo-1106 · Davinci accuracy | accuracy: 0.53 -> 0.57 | Helped |
| use an incorrect persona distribution (ablation) | IQA-Eval · MMLU interaction meta-evaluation | Pearson correlation (Fluency): 0.787; 0.741; 0.779; 0.734; 0.703 | Mixed |
| evaluate on interactive question answering rather than direct answers | IQA-Eval with GPT-4 evaluator · Natural Questions | accuracy: 0.38 -> 0.44; 0.34 -> 0.40; 0.30 -> 0.32; 0.24 -> 0.36 | Helped |
Surprising findings
- Interaction-based helpfulness and fluency rankings of IQA models differ from their non-interactive accuracy rankings; e.g., Claude leads helpfulness on HotpotQA while GPT-4 leads accuracy.
- LEA evaluations of human-model interactions correlate only moderately (0.306 helpfulness for ChatGPT) with humans, far below the around 0.6 seen for LEA-LEA interactions.
Limitations
- Evaluation restricted to multiple-choice QA on MMLU for meta-evaluation; generalization to other QA formats untested.
- Strong self-enhancement bias concerns when the LEA shares the model family with the IQA model; debiasing only partly mitigates it.
- Persona assignments are sensitive to incorrect distributions and persona designs could be biased.
Experimental setup
Datasets: MMLU HotpotQA AmbigQA Natural Questions RealToxicityPrompts
Benchmarks: MMLU human-model interaction data HotpotQA AmbigQA Natural Questions
Metrics: Pearson correlation helpfulness (5-point Likert) fluency (5-point Likert) number of queries accuracy success rate on offensiveness/harmfulness
Baselines: Text-Davinci-001 Text-Babbage-001 Davinci-001 GPT-3.5-turbo GPT-4 Claude-1 Llama-2-7B Zephyr-alpha
Builds on: G-Eval peer discussion prompting
Closest papers by meaning
- ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate (ICLR 2024 · similarity 0.90)
- Rethinking Generative Large Language Model Evaluation for Semantic Comprehension (ICML 2024 · similarity 0.89)
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (ICML 2024 · similarity 0.88)
- A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators (AAAI 2024 · similarity 0.87)
- Unbiased Evaluation of Large Language Models from a Causal Perspective (ICML 2025 · similarity 0.87)
- Putting People in LLMs’ Shoes: Generating Better Answers via Question Rewriter (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.