Research map / Large language models / Language, safety and interpretability
Recording for Eyes, Not Echoing to Ears: Contextualized Spoken-to-Written Conversion of ASR Transcripts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Contextualized Spoken-to-Written conversion (CoS2W) of ASR transcripts, releases the document-level SWAB benchmark (Chinese/English meetings, podcasts, lectures), and evaluates LLMs with context and auxiliary-information methods, using GPT-4 as a judge validated against humans.
Problem
ASR transcripts contain recognition errors, disfluencies, ungrammatical and incomplete sentences, giving poor readability and hurting downstream tasks; no benchmark or study exists for converting whole documents from spoken to written style using context and auxiliary information.
Approach
Frame spoken-to-written conversion as a contextualized, document/paragraph-level task that jointly corrects ASR errors, grammar, and informal style, and exploit local/global context and auxiliary information (summaries, keywords) via in-context learning; validate LLM-as-evaluator for faithfulness and formality against human rankings.
Key results
- GPT-4 at chunk level achieves BLEURT 62.15 and S-Faithful 7.32 on SWAB, outperforming QWen-Max, QWen-14B and LLaMA3-8B.
- Providing keyword-based auxiliary information raises GPT-4 CK-Recall from 22.49% to 45.42%.
- LLM evaluator faithfulness rankings correlate with human rankings at Spearman 0.75; formality at 0.61.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply CoS2W before machine translation | CSANMT · BSTC development set | BLEURT: 57.07 to 61.9 | Helped |
| use paragraph-level rather than chunk-level processing (ablation) | GPT-4 · SWAB | BLEURT: 62.15 -> 63.32 | Helped |
| use paragraph-level rather than chunk-level processing (ablation) | GPT-4 · SWAB | CK-Recall: 22.49 -> 19.20 | Hurt |
| process the entire document in one input rather than in chunks (ablation) | GPT-4 · SWAB | BLEURT: 62.15 -> 10.64 | Hurt |
| add local context (ablation) | GPT-4, paragraph-level CoS2W · SWAB | CK-Recall: 19.20% to 22.21% | Helped |
| add global context (ablation) | GPT-4, paragraph-level CoS2W · SWAB | BLEURT: 63.32 -> 66.00 | Helped |
| add both local and global context (ablation) | GPT-4, paragraph-level CoS2W · SWAB | CK-Recall: 19.20 -> 23.78 | Helped |
| provide original auxiliary information (ablation) | GPT-4, chunk-level CoS2W · SWAB | CK-Recall: 22.49% to 43.70% | Helped |
| retrieve top-10 relevant auxiliary sentences (ablation) | GPT-4, chunk-level CoS2W · SWAB | CK-Recall: 22.49 -> 37.11 | Helped |
| provide an LLM-generated summary of auxiliary information (ablation) | GPT-4, chunk-level CoS2W · SWAB | BLEURT: 62.15 -> 62.75 | Helped |
| provide an LLM-generated summary of auxiliary information (ablation) | GPT-4, chunk-level CoS2W · SWAB | CK-Recall: 22.49 -> 40.40 | Helped |
| provide LLM-extracted keywords from auxiliary information (ablation) | GPT-4, chunk-level CoS2W · SWAB | CK-Recall: 22.49 -> 45.42 | Helped |
Surprising findings
- Document-level CoS2W is much worse than chunk or paragraph level, with 76.10% of paragraphs left blank by GPT-4 due to alignment failures.
- Smaller open models like QWen-14B have high formality on non-blank paragraphs (8.29) but low overall scores due to instruction-following failures.
Limitations
- SWAB has only a single reference target per transcript, limiting reliability of BLEU/ROUGE.
- Initial targets are produced by GPT-4 and GPT-4 is also used as evaluator, introducing bias.
- Faithfulness remains a problem for LLMs, including paragraph drift and fabrication.
Experimental setup
Datasets: AliMeeting AMI YouTube podcasts and lectures (self-collected) FunASR transcripts
Benchmarks: SWAB
Metrics: BLEU ROUGE-L BLEURT CK-Recall S-Faithful S-Formal blank paragraph rate Spearman correlation
Baselines: GPT-4 QWen-Max QWen-14B LLaMA3-8B RoBERTa formality classifier
Builds on: BM25 BLEURT Paraformer
Closest papers by meaning
- Speech Recognition Meets Large Language Model: Benchmarking, Models, and Exploration (AAAI 2025 · similarity 0.86)
- BIGOS V2 Benchmark for Polish ASR: Curated Datasets and Tools for Reproducible Evaluation (NeurIPS 2024 · similarity 0.85)
- SwitchLingua: The First Large-Scale Multilingual and Multi-Ethnic Code-Switching Dataset (NeurIPS 2025 · similarity 0.85)
- LiveCC: Learning Video LLM with Streaming Speech Transcription at Scale (CVPR 2025 · similarity 0.85)
- WorldSpeech: A Multilingual Speech Corpus from Around the World (NeurIPS 2026 · similarity 0.84)
- Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.