Research map / Large language models / Retrieval and knowledge
Towards Verifiable Text Generation with Generative Agent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes R2-MGA, a memory-augmented generative agent that retrieves and reflects on past experiences to build best-matched in-context demonstrations and iteratively assesses and refines answers for verifiable text generation with citations on ALCE.
Problem
One-step text generation with citations (ALCE benchmark) uses the same in-context demonstrations for all questions regardless of question type and cannot refine answers after the first generation, limiting answer correctness and citation quality.
Approach
A memory bank stores full-generation, assessment, and planning histories; for each module it retrieves the most similar memory snippet via embedding similarity, reflects on it to produce a rationale with gpt-3.5-turbo, and uses the snippet plus rationale as a 1-shot demonstration, followed by assess-plan-refine answer refinement.
Key results
- On ELI5, R2-MGA gains up to +58.8% relative correctness, +139.3% citation recall, and +154.7% citation precision over one-step baselines.
- LLaMA-70B-powered R2-MGA consistently outperforms the ChatGPT-based ALCE-base across datasets and prompting strategies and beats GPT-4 ALCE-base in 6 of 10 VANILLA cases.
- Ablations show removing answer refinement causes the largest drops (up to -16.6 citation recall on ELI5 for Vicuna-13B), with best-matched demonstrations and rationales also contributing.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| R2-MGA rather than ALCE-CoT | Vicuna-13B text generation with citations · ASQA | Correctness: 36.4 -> 37.1 | Helped |
| add the best-matched demonstration instead of always using the same demonstration (ablation) | R2-MGA with Vicuna-13B · ELI5 | Citation Recall: 38.9 -> 28.2 (-10.7) | Helped |
| add answer refinement with Assessment and Planning & Action modules (ablation) | R2-MGA with Vicuna-13B · ELI5 | Citation Recall: 38.9 -> 22.3 (-16.6) | Helped |
| include reasoning rationales in the demonstration (ablation) | R2-MGA with Vicuna-13B · ELI5 | Citation Recall: 38.9 -> 35.9 (-3.0) | Helped |
Surprising findings
- The one-time action execution paradigm consistently outperforms step-by-step execution, as step-by-step refinement can introduce new flaws that later actions cannot fix.
Limitations
- Requires additional LLM calls for assessment, planning, action, and rationale generation (1.42x to 2.04x compute over ALCE-base).
- Rationales are generated by gpt-3.5-turbo-0125, adding dependence on a closed-source model.
- Gains from memory depend on accumulating snippets; the memory is built sequentially over the same benchmark entries.
Experimental setup
Datasets: ASQA QAMPARI ELI5
Benchmarks: ALCE (ASQA, QAMPARI, ELI5)
Metrics: correctness citation recall citation precision correctness recall-5
Baselines: ALCE-base ALCE-CoT SELF-REFINE VTG A2R
Builds on: ALCE Reflexion-style reflection SELF-REFINE Chain-of-Thought Generative Agents
Closest papers by meaning
- Retrieval is Accurate Generation (ICLR 2024 · similarity 0.89)
- Precise Information Control in Long-Form Text Generation (NeurIPS 2025 · similarity 0.88)
- Reflection-Window Decoding: Text Generation with Selective Refinement (ICML 2025 · similarity 0.86)
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (ICLR 2024 · similarity 0.86)
- Guaranteed Generation from Large Language Models (ICLR 2025 · similarity 0.85)
- DataGen: Unified Synthetic Dataset Generation via Large Language Models (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.