Research map / Large language models / Retrieval and knowledge

On the Vulnerability of Applying Retrieval-Augmented Generation within Knowledge-Intensive Application Domains

ICML 2025 · Large language models › Retrieval and knowledge · approach: RAG robustness and evaluation

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows dense retrievers used in medical and legal RAG are vulnerable to universal poisoning attacks (query-appended documents) and proposes a Mahalanobis-style shrinkage-covariance detection defense that achieves near-perfect detection.

Problem

Retrieval systems in RAG for knowledge-intensive domains (medical, legal) can be poisoned so that attacker-specified documents are retrieved for many users, and existing defenses such as embedding l2-norm detection fail.

Approach

Attackers append target information to queries to create poisoned documents; an "orthogonal augmentation" property explains why the query-document similarity is preserved, and a shrinkage-regularized Mahalanobis distance over anchor clean-document embeddings detects poisoned documents where l2 norm fails.

Key results

• Across 225 medical attack setups (3 corpora, 3 retrievers, 5 query sets, 5 target types), top-2 attack success rates are mostly 0.6-1.0.
• The proposed shrinkage-Mahalanobis detector achieves over 98% detection rate on the 225 attack cases, while l2-norm detection is below 50% for MedCPT and Specter.
• Attacks remain effective under query paraphrasing, with top-2 success around 0.8 for most cases.

Measured effects reported in the paper

Surprising findings

• Clean retrieved documents are nearly orthogonal to their queries (about 70 degrees for Contriever), so poisoned documents that keep high similarity to queries stand out only under distribution-aware distances.
• Attacks succeed even when the attacker knows only paraphrased versions of the queries.

Limitations

• PubMed corpus was a random ~2M-document subset due to limited compute.
• Attacker is assumed to know (or closely approximate) target queries and to be able to inject documents into the corpus.
• Detection relies on a defender-held anchor set of clean documents and a cross-validated shrinkage parameter beta.

Experimental setup

Datasets: MMLU-Med MedQA-US MedMCQA PubMedQA BioASQ-Y/N Textbooks StatPearls PubMed ALQA MS MARCO Natural Questions HotpotQA

Benchmarks: MMLU-Med MedQA-US MedMCQA PubMedQA BioASQ-Y/N ALQA

Metrics: top-2 retrieval success rate top-1 retrieval success rate detection rate filtering rate

Baselines: l2-norm embedding defense PoisonedRAG

Builds on: PoisonedRAG MedRAG benchmark Contriever MedCPT

Closest papers by meaning

• Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems (ICLR 2025 · similarity 0.91)
• ReliabilityRAG: Effective and Provably Robust Defense for RAG-based Web-Search (NeurIPS 2025 · similarity 0.89)
• SeCon-RAG: A Two-Stage Semantic Filtering and Conflict-Free Framework for Trustworthy RAG (NeurIPS 2025 · similarity 0.88)
• RAGGED: Towards Informed Design of Scalable and Stable RAG Systems (ICML 2025 · similarity 0.88)
• Worse than Zero-shot? A Fact-Checking Dataset for Evaluating the Robustness of RAG Against Misleading Retrievals (NeurIPS 2025 · similarity 0.88)
• Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting (ICLR 2025 · similarity 0.87)