Research map / Large language models / Retrieval and knowledge

Toward Verifiable Instruction-Following Alignment for Retrieval Augmented Generation

AAAI 2025 · Large language models › Retrieval and knowledge · approach: RAG robustness and evaluation

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VIF-RAG, a verifiable synthetic pipeline that builds a >100K instruction-following RAG training set from under 100 hand-written atomic instructions using code-executor verification, and introduces the FollowRAG benchmark (~2.8K samples).

Problem

Instruction-following alignment of LLMs in retrieval-augmented generation (RAG) settings is under-studied; existing IF data and benchmarks do not transfer well to RAG, and SFT on general data can hurt RAG-side instruction following.

Approach

Synthesize complex instructions from a minimal set of atomic constraints, rewrite them with a strong LLM, verify each instruction by having the LLM write verification functions and test cases run in a Python executor, then combine with RAG QA queries and retrieved passages and filter responses by executor checks and LLM-judged query-instruction consistency.

Key results

• On FollowRAG, Mistral-7B-SFT-VIF-RAG reaches an ALL IF score of 47.4 versus 19.3 for Mistral-7B-SFT and 38.4 for Evol-Instruct-based SFT (prompt-level values as reported in the main table).
• On IFEval, Qwen1.5-14B-SFT-VIF-RAG reaches 46.3 (strict prompt) and 60.0 (strict instruction) vs. 38.4 and 49.4 for its SFT version.
• VIF-RAG preserves foundational abilities; e.g., Qwen1.5-7B-SFT-VIF-RAG scores 61.4 on GSM8K vs 48.6 for its SFT version.

Measured effects reported in the paper

Surprising findings

• Standard SFT on general data such as ShareGPT lowers FollowRAG instruction-following scores relative to base models (e.g., Mistral NQ-IF 25.7 to 21.0), even though it improves IFEval.
• Removing executor-based verification causes the largest drop in FollowRAG IF performance in the ablation.

Limitations

• Verification relies on GPT-4-class models for rewriting, verification code, and consistency scoring, so pipeline cost and dependence on proprietary models are unaddressed.
• RAG scores in FollowRAG are judged by GPT-4o and only 100 samples per dataset were human-checked for agreement.
• The benchmark covers 22 atomic constraint types, which may not reflect all real-world instruction types.

Experimental setup

Datasets: Natural Questions TriviaQA HotpotQA WebQuestionsSP ShareGPT Wikipedia (DPR corpus) Freebase

Benchmarks: FollowRAG IFEval FollowBench MT-Bench Arena-Hard C-Eval MMLU GSM8K HumanEval

Metrics: instruction-level accuracy (strict and loose) prompt-level accuracy RAG score (0/0.5/1) FollowBench SSR average Pass@1

Baselines: Conifer Evol-Instruct Deita-7B-V1.0-SFT SFT version of each backbone

Builds on: IFEval FollowBench Evol-Instruct Conifer DPR Deita

Closest papers by meaning

• Retrieval is Not Enough: Enhancing RAG through Test-Time Critique and Optimization (NeurIPS 2025 · similarity 0.91)
• SelfCodeAlign: Self-Alignment for Code Generation (NeurIPS 2024 · similarity 0.89)
• Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems (ICLR 2025 · similarity 0.89)
• Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting (ICLR 2025 · similarity 0.89)
• Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models (ICLR 2025 · similarity 0.88)
• RAG-DDR: Optimizing Retrieval-Augmented Generation Using Differentiable Data Rewards (ICLR 2025 · similarity 0.88)