Research map / Large language models / Language, safety and interpretability
Characterizing Information Seeking Events in Health-Related Social Discourse
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TREAT-ISE, a 5,083-post multilabel Reddit dataset of expert-defined treatment information-seeking events for opioid use disorder medications, and benchmarks ML, transformer, and ChatGPT classifiers on it.
Problem
Event-level analysis of health information-seeking in social media discourse is lacking, especially for medications for opioid use disorder (MOUD), where no annotated event dataset exists.
Approach
Defines five expert-guided information-seeking event categories for MOUD recovery and frames event identification on Reddit posts as multilabel, multiclass classification, benchmarking classic, transformer and ChatGPT prompting approaches.
Key results
- XLNet achieved the best weighted F1 of 0.774 among all models on TREAT-ISE, with the paper's abstract reporting a 77.4% F1 benchmark.
- ChatGPT with chain-of-thought prompting reached a weighted F1 of 0.631, below all fine-tuned transformer models, and showed much higher recall than precision.
- ChatGPT's average overprediction ratio was about 45%, versus substantially lower rates for XLNet in most classes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a long rather than short prompt in zero-shot classification (ablation) | ChatGPT · TREAT-ISE test | weighted F1: 0.433 -> 0.581 | Helped |
| Use a long rather than short prompt in few-shot classification (ablation) | ChatGPT · TREAT-ISE test | weighted F1: 0.609 -> 0.620 | Helped |
| Add few-shot examples using a short prompt instead of zero-shot short prompting (ablation) | ChatGPT · TREAT-ISE test | weighted F1: 0.433 -> 0.609 | Helped |
| Add few-shot examples using a long prompt instead of zero-shot long prompting (ablation) | ChatGPT · TREAT-ISE test | weighted F1: 0.581 -> 0.620 | Helped |
| Use chain-of-thought prompting instead of few-shot long prompting (ablation) | ChatGPT · TREAT-ISE test | weighted F1: 0.620 -> 0.631 | Helped |
| Use XLNet rather than ChatGPT with chain-of-thought prompting | TREAT-ISE classifiers · TREAT-ISE test | weighted F1: 0.631 -> 0.774 | Helped |
Surprising findings
- Zero- and few-shot ChatGPT variants underperformed simple baselines such as BiGRU and fine-tuned BERT variants on this domain-specific task.
- ChatGPT frequently mislabeled posts containing dosage information or withdrawal symptoms as treatment-taking or psychophysical-effect events, even when they did not seek information.
Limitations
- Dataset restricted to a single subreddit (r/Suboxone), possibly biasing results toward that community.
- Relatively small dataset of 5,083 samples due to costly expert-guided annotation; crowd-sourced annotation had only 40.5% agreement.
- Only gpt-3.5-turbo was evaluated for LLM performance; newer models such as GPT-4 and LLaMA were not tested.
Experimental setup
Datasets: TREAT-ISE
Benchmarks: TREAT-ISE
Metrics: precision recall F1 weighted F1 Cohen's kappa McNemar's test
Baselines: Logistic Regression NBSVM FastText BiGRU BERT RoBERTa DistilBERT ELECTRA XLNet MPNet ChatGPT
Builds on: FastText BERT XLNet RoBERTa
Closest papers by meaning
- End-To-End Causal Effect Estimation from Unstructured Natural Language Data (NeurIPS 2024 · similarity 0.81)
- Position: Retrieval-augmented systems can be dangerous medical communicators (ICML 2025 · similarity 0.81)
- Reasoning-Enhanced Healthcare Predictions with Knowledge Graph Community Retrieval (ICLR 2025 · similarity 0.80)
- MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records (AAAI 2024 · similarity 0.80)
- One Loss to Rule Them All: Marked Time-to-Event for Structured EHR Foundation Models (NeurIPS 2026 · similarity 0.80)
- A Joint Framework with Heterogeneous-Relation-Aware Graph and Multi-Channel Label Enhancing Strategy for Event Causality Extraction (AAAI 2024 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.