Research map / Large language models / Retrieval and knowledge
Position: Retrieval-augmented systems can be dangerous medical communicators
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper arguing that retrieval-augmented medical answers can mislead patients even when factually accurate, supported by a query study of Google AI Overview and Perplexity and calls for pragmatically aligned systems.
Problem
Retrieval-augmented generation and citation grounding are assumed to make medical answers accurate, but literally accurate RAG outputs can still convey misleading overall messages to patients.
Approach
Frames RAG medical failures through pragmatics and rational communication theory, identifying narrow literal interpretation of queries, decontextualization of sources, and lack of reasoning about downstream patient interpretation, and proposes pragmatic benchmarks and system designs.
Key results
- Google AI Overview and Perplexity mention disputed status of conditions in 100% of direct queries but only 56% (Google) and 69% (Perplexity) of presupposing symptom queries.
- Source citation Jaccard similarity between 'why dangerous' and 'why safe' procedure queries was 0.16 for Google AI Overview and 0.31 for Perplexity.
- Complication queries mention rarity statistics for only 4% (Google) and 5% (Perplexity) of responses and benefits for 6% and 10%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use symptom queries that presuppose a disputed condition exists instead of directly querying the condition | Google AI Overview · 13 disputed conditions | responses mentioning controversy: 100% -> 56% | Hurt |
| Use symptom queries that presuppose a disputed condition exists instead of directly querying the condition | Perplexity AI · 13 disputed conditions | responses mentioning controversy: 100% -> 69% | Hurt |
| Ask why a procedure is safe rather than dangerous | Google AI Overview · 28 procedure safety queries | average Jaccard similarity of cited webpages: 0.16 | Mixed |
| Ask why a procedure is safe rather than dangerous | Perplexity AI · 28 procedure safety queries | average Jaccard similarity of cited webpages: 0.31 | Mixed |
| Answer queries about procedure complications | Google AI Overview · 28 procedures | responses mentioning statistics on complication rarity: 4% | Hurt |
| Answer queries about procedure complications | Google AI Overview · 28 procedures | responses mentioning benefits: 6% | Hurt |
| Answer queries about procedure complications | Perplexity AI · 28 procedures | responses mentioning statistics on complication rarity: 5% | Hurt |
| Answer queries about procedure complications | Perplexity AI · 28 procedures | responses mentioning benefits: 10% | Hurt |
Surprising findings
- Responses to opposing-valence safety queries often cite the same source yet extract very different facts.
- Presupposing symptom queries lead to far fewer mentions of scientific controversy than direct queries.
Limitations
- Query analysis is limited to 178 template-generated queries on two engines over a few days.
- LLM-as-judge labels, with manual adjudication only for low-confidence disagreements.
- Does not measure actual user belief change; human impact is argued rather than tested.
Experimental setup
Datasets: procedurally generated medical queries (178 queries over 13 disputed conditions and 28 procedures)
Benchmarks: procedurally generated medical query set (178 queries)
Metrics: percentage of responses mentioning controversy percentage mentioning complication statistics percentage mentioning benefits Jaccard similarity of cited sources
Baselines: Google AI Overview Perplexity AI
Builds on: Rational Speech Acts
Closest papers by meaning
- Deceptive Grounding: Entity Attribution Failure in Clinical Retrieval-Augmented Generation (NeurIPS 2026 · similarity 0.87)
- On the Vulnerability of Applying Retrieval-Augmented Generation within Knowledge-Intensive Application Domains (ICML 2025 · similarity 0.83)
- MedVIGIL: Evaluating Trustworthy Medical VLMs Under Broken Visual Evidence (NeurIPS 2026 · similarity 0.83)
- Reasoning-Enhanced Healthcare Predictions with Knowledge Graph Community Retrieval (ICLR 2025 · similarity 0.82)
- DoAtlas-1: A Causal Compilation Paradigm for Clinical AI (NeurIPS 2026 · similarity 0.82)
- RAD: Towards Trustworthy Retrieval-Augmented Multi-modal Clinical Diagnosis (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.