Research map / Large language models / Language, safety and interpretability

Entity Only vs. Inline Approaches: Evaluating LLMs for Adverse Drug Event Detection in Clinical Text (Student Abstract)

AAAI 2025 · Large language models › Language, safety and interpretability · approach: Data and hallucination detection · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Compares inline entity-marking versus entity-only extraction prompting for zero- and few-shot adverse drug event and drug detection on the n2c2 2018 clinical notes using Llama-3.1 8B, finding entity-only extraction performs better.

Problem

Detecting adverse drug events (ADEs) and drugs in unstructured clinical discharge summaries (n2c2 2018) without ADE-specific training data, and identifying which LLM prompting format works best.

Approach

Contrasts an inline approach (inserting markers around entity spans in the text) with an entity-only approach (listing extracted entities separately) for prompting an LLM on ADE detection, varying the number of in-context examples from 0 to 10.

Key results

• Zero-shot entity-only extraction achieved the best ADE F1 (0.115) and drug F1 (0.525), versus 0.062 ADE F1 and 0.351 drug F1 for zero-shot inline.
• Entity-only outperformed inline on all metrics at most example counts, but increasing examples from 0 to 10 did not consistently improve performance.
• Models over-predicted and repeated entities, lowering precision while keeping recall high.

Measured effects reported in the paper

Surprising findings

• Zero-shot prompting performed best for both drug and ADE F1, and adding more examples did not consistently help.
• Absolute ADE F1 scores remained very low (at most 0.115), despite high recall for drugs.

Limitations

• Only one relatively small LLM (Llama-3.1 8B) was evaluated.
• Absolute ADE F1 scores are low, leaving substantial room for improvement.
• Default generation parameters were used without tuning.

Experimental setup

Datasets: n2c2 2018 ADE task (505 discharge summaries from MIMIC-III)

Benchmarks: n2c2 2018 ADE test set

Metrics: lenient F1 precision recall

Baselines: inline prompting

Closest papers by meaning

• MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey (NeurIPS 2024 · similarity 0.85)
• SafetyPrompts: A Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety (AAAI 2025 · similarity 0.85)
• MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models (NeurIPS 2024 · similarity 0.85)
• CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMs (NeurIPS 2025 · similarity 0.85)
• ER-Reason: A Benchmark Dataset for LLM Clinical Reasoning in the Emergency Room (NeurIPS 2026 · similarity 0.84)
• RTP-LX: Can LLMs Evaluate Toxicity in Multilingual Scenarios? (AAAI 2025 · similarity 0.84)