Research map / Large language models / Language, safety and interpretability

MEDSAGE: Enhancing Robustness of Medical Dialogue Summarization to ASR Errors with LLM-generated Synthetic Dialogues

AAAI 2025 · Large language models › Language, safety and interpretability · approach: Speech and audio language models · 6 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Generates ASR-like noisy clinical dialogues with LLMs via in-context learning and a word-level error-tagging syntax calibrated to real ASR error profiles, then uses them to augment summarization fine-tuning.

Problem

ASR errors degrade downstream clinical dialogue summarization, but medical audio and paired ASR transcripts are scarce, so conventional noise-robust augmentation is infeasible and ASR post-correction via prompting is unreliable for small LLMs.

Approach

Prompt LLMs with a few paired clean/ASR medical dialogue examples plus a tagging syntax that marks which words to corrupt and how, with tag rates sampled from the target ASR model's measured WER and error-type distribution; the resulting synthetic noisy dialogues augment summarization fine-tuning.

Key results

• Augmenting training with MEDSAGE synthetic noisy dialogues improves DSE F1 by up to 16.4% over zero-shot on ASR-transcribed Primock57 test set.
• Whisper-large ASR (25% WER) gives summarization close to ground-truth transcripts, while Wav2vec2-base (45% WER) drops entity F1 from 22.11 to 16.99 with Llama-3-8B.
• Denoising ASR transcripts with Llama-3-8B consistently hurt summarization across all three summarizers.

Measured effects reported in the paper

Surprising findings

• Prompt-based denoising of ASR transcripts with an 8B LLM consistently degraded summarization quality rather than helping.
• Fine-tuning on clean dialogues alone gave only marginal gains or even degraded performance depending on the summarizer (e.g., Mistral-7B F1 dropped from 20.77 zero-shot to 16.76).

Limitations

• Evaluation limited to a small test set of 57 enacted dialogues (Primock57) and a single synthetic training corpus.
• Denoising baseline only tested with an 8B model, so the claim that LLM post-correction requires >100B parameters is not directly tested here.
• Synthetic error generation relies on a few in-context examples from one dataset and on the measured error profile of specific ASR models.

Experimental setup

Datasets: Primock57 NoteChat-1000

Benchmarks: Primock57

Metrics: domain-specific entity F1 ROUGE-L BERTScore WER

Baselines: Zero-shot FT on clean Denoising with Llama-3-8B Random corruption augmentation

Builds on: in-context learning LoRA Wagner-Fisher alignment

Closest papers by meaning

• MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey (NeurIPS 2024 · similarity 0.85)
• Recording for Eyes, Not Echoing to Ears: Contextualized Spoken-to-Written Conversion of ASR Transcripts (AAAI 2025 · similarity 0.84)
• Scaling Speech-Text Pre-training with Synthetic Interleaved Data (ICLR 2025 · similarity 0.84)
• MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records (AAAI 2024 · similarity 0.83)
• Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-World Multi-Turn Dialogue (AAAI 2024 · similarity 0.83)
• From Medical Records to Diagnostic Dialogues: A Clinical-Grounded Approach and Dataset for Psychiatric Comorbidity (ICLR 2026 · similarity 0.83)