Research map / Large language models / Retrieval and knowledge
EvalQAG: A Framework for Automatic Complex QA Generation and a Benchmark QA Dataset for Policy Documents
Kirtan Brijeshbhai Soni, Krish Rupapara, Arpit Rana, Ghanshyam Verma, Paul Buitelaar
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune LLaMA3-8B on 10,000 REPolicyQA pairs | LLaMA3-8B · REPolicyQA test, all questions (with context) | F1: 0.26 -> 0.64 | Improved |
| fine-tune LLaMA3-8B on REPolicyQA, no context given | LLaMA3-8B · REPolicyQA test, all questions (no context) | F1: 0.18 -> 0.41 | Improved |
| fine-tune on REPolicyQA for Legal Obligation questions | LLaMA3-8B · REPolicyQA Legal Obligation | F1: 0.16 -> 0.50 | Improved |
| fine-tune on REPolicyQA for Yes-No questions | LLaMA3-8B · REPolicyQA Yes-No | F1: 0.51 -> 0.85 | Improved |
| add local (semantic dedup) filtering (ablation) | EvalQAG pipeline · REPolicyQA generated QA quality | mean LLM-judge score: 8.32 -> 8.36 | No clear effect |
| add global score-threshold filtering after local filtering (ablation) | EvalQAG pipeline · REPolicyQA generated QA quality | mean LLM-judge score: 8.36 -> 9.06 | Improved |
| two-stage filtering effect on Relevance score (ablation) | EvalQAG pipeline · REPolicyQA generated QA quality | Relevance: 6.70 -> 8.46 | Improved |
Closest papers by meaning
- ClimaQA: An Automated Evaluation Framework for Climate Question Answering Models (ICLR 2025 · similarity 0.87)
- FailureSensorIQ: A Multi-Choice QA Dataset for Understanding Sensor Relationships and Failure Modes (NeurIPS 2025 · similarity 0.82)
- CofCA: A STEP-WISE Counterfactual Multi-hop QA benchmark (ICLR 2025 · similarity 0.82)
- SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs (ICLR 2024 · similarity 0.82)
- Assessing Large Language Models on Climate Information (ICML 2024 · similarity 0.82)
- A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering (NeurIPS 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
