Research map / Large language models / Retrieval and knowledge
BAND: Biomedical Alert News Dataset
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces BAND, a corpus of 1,508 expert-selected ProMED-style biomedical outbreak alert news items annotated with 30 epidemiology/event/ethics questions, and benchmarks NER, QA and event extraction with fine-tuned and zero-shot LLMs.
Problem
Existing disease surveillance systems and datasets focus on outbreak statistics (location, disease names) and lack well-annotated epidemiological analysis of outbreak news alerts that requires expert reasoning.
Approach
A newly annotated dataset of outbreak alert news with 30 expert-designed epidemiology questions requiring inference beyond the text (e.g., inferring country from city), plus benchmark tasks for NER, abstractive QA and event extraction comparing fine-tuned small LMs against zero-shot ChatGPT.
Key results
- NER: TokenBased and SpanBased reach F1 0.715 on the stratified split, while ChatGPT reaches 0.363.
- QA: fine-tuned BLOOM (560M) achieves 0.735 random and 0.751 stratified accuracy versus 0.497 and 0.413 for zero-shot ChatGPT.
- Event extraction: T5 and BART reach overall F1 of about 60.9 on random split, versus 47.7 for ChatGPT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on BAND annotations instead of using zero-shot ChatGPT | NER models · BAND NER Random split | F1-score: 0.339 vs 0.660–0.642 | Helped |
| Train on BAND annotations instead of using zero-shot ChatGPT | NER models · BAND NER Stratified split | F1-score: 0.363 vs 0.715 | Helped |
| Fine-tune BLOOM rather than use zero-shot ChatGPT | QA models · BAND QA Rand split | accuracy: 0.497 -> 0.735 | Helped |
| Fine-tune BLOOM rather than use zero-shot ChatGPT | QA models · BAND QA Stratified split | accuracy: 0.413 -> 0.751 | Helped |
| Use an encoder-decoder model instead of a decoder-only model | Event extraction models · BAND EE Random split | Overall F1: GPTNEO 34.34; T5 60.88; Bart 60.86 | Helped |
| Use zero-shot ChatGPT instead of fine-tuned GPT2 for event extraction | Event extraction models · BAND EE Random split | Overall F1: 45.34 -> 47.71 | Helped |
Surprising findings
- ChatGPT often answers 'Cannot Infer' instead of performing the background-knowledge inference the questions require.
- Larger fine-tuned models (e.g., bigger GPT-style models) tended to answer in their own words rather than the pre-defined format, so the smallest Galactica (125M) was used.
Limitations
- Dataset is drawn mainly from ProMED-mail, which may bias toward certain outbreak types and regions.
- Annotation is subjective and relies on annotator judgment, with residual inconsistency.
- Dataset is predominantly English-language news.
Experimental setup
Datasets: ProMED-mail alerts
Benchmarks: BAND (random split) BAND (stratified split)
Metrics: precision recall F1-score exact match accuracy
Baselines: CRFBased TokenBased SpanBased ChatGPT T5 Bart GPT2 OPT GPTNEO Galactica BLOOM
Builds on: T5 BART OPT Galactica BLOOM
Closest papers by meaning
- Data Monitoring for Large Scale Public Health Data (AAAI 2025 · similarity 0.80)
- Evaluating AI-based Scientific Knowledge Synthesis with Epidemiological Systematic Reviews (NeurIPS 2026 · similarity 0.79)
- Characterizing Information Seeking Events in Health-Related Social Discourse (AAAI 2024 · similarity 0.79)
- Aligning Time-series by Local Trends: Applications in Public Health (AAAI 2025 · similarity 0.78)
- Entity Only vs. Inline Approaches: Evaluating LLMs for Adverse Drug Event Detection in Clinical Text (Student Abstract) (AAAI 2025 · similarity 0.78)
- ACES: Automatic Cohort Extraction System for Event-Stream Datasets (ICLR 2025 · similarity 0.78)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.