Research map / Large language models / Language, safety and interpretability
MathSpeech: Leveraging Small LMs for Accurate Conversion in Mathematical Speech-to-Formula
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MathSpeech, a two-stage pipeline in which two fine-tuned T5-small models (ASR error corrector plus LaTeX translator, trained end-to-end) convert ASR outputs of spoken math into LaTeX, outperforming GPT-4o and Gemini-Pro few-shot on a new MIT OCW math speech benchmark.
Problem
ASR systems produce verbose, error-prone textual transcriptions of spoken mathematical expressions instead of LaTeX, hindering subtitles for math lectures.
Approach
Chain an ASR error corrector and a LaTeX translator, both T5-small (~120M total), and train them end-to-end with a weighted cross-entropy loss; the corrector takes top-1 outputs from multiple different ASR models as input, and training data is created by TTS-then-ASR to capture realistic ASR errors.
Key results
- MathSpeech (2x T5-small, 120M) reduces CER from 0.390 (GPT-4o, averaged over ASR inputs) to 0.298 and achieves higher ROUGE and BLEU than GPT-4o on average (BLEU 0.689 vs 0.505).
- With whisper-small & whisper-base ASR inputs, MathSpeech reaches CER 0.243 versus 0.379 for GPT-4o.
- MathSpeech reports 0.45 s latency for a 5-second clip on a single V100, versus 0.65 s for GPT-4o via API.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MathSpeech instead of GPT-4o | LaTeX translation from two ASR outputs · MathSpeech benchmark, two-ASR total average | CER: 0.390 -> 0.298 | Helped |
| Use MathSpeech instead of GPT-4o | LaTeX translation from two ASR outputs · MathSpeech benchmark, two-ASR total average | ROUGE-1: 0.838 -> 0.845 | Helped |
| Use MathSpeech instead of GPT-4o | LaTeX translation from two ASR outputs · MathSpeech benchmark, two-ASR total average | ROUGE-L: 0.833 -> 0.840 | Helped |
| Use MathSpeech instead of GPT-4o | LaTeX translation from two ASR outputs · MathSpeech benchmark, two-ASR total average | BLEU: 0.505 -> 0.689 | Helped |
| Add an Error Corrector before the LaTeX Translator (ablation) | MathSpeech · MathSpeech benchmark, Whisper-base | CER: 0.693 -> 0.336 | Helped |
| Add an Error Corrector before the LaTeX Translator (ablation) | MathSpeech · MathSpeech benchmark, Whisper-small | CER: 0.630 -> 0.309 | Helped |
| Train the corrector and translator end-to-end instead of separately connecting them (ablation) | Two-stage T5-small pipeline · MathSpeech benchmark, Whisper-small | CER: 0.357 -> 0.243 | Helped |
| Use a two-stage corrector-and-translator structure instead of one T5 model (ablation) | MathSpeech · MathSpeech benchmark, Whisper-small | CER | Helped |
Surprising findings
- Off-the-shelf ASR models (Whisper, Canary) produce high WER on formula speech (about 29-35%) despite strong leaderboard scores, partly due to label ambiguity such as 'one' vs '1'.
- The 120M MathSpeech pipeline generalizes to Whisper-largeV3, an ASR model not used in corrector training.
Limitations
- Metrics such as BLEU and CER are imperfect for LaTeX, since many LaTeX strings represent the same formula.
- Does not address detecting formula segments within general speech or completing interrupted formulas.
- Baseline LLMs used only 1-shot prompting, and the evaluation is limited to one benchmark built from MIT OCW lecture audio.
Experimental setup
Datasets: MathBridge (23M Spoken English-LaTeX pairs) TTS-generated ASR error corpus (VITS + Whisper/Canary)
Benchmarks: MathSpeech benchmark (1,101 MIT OpenCourseWare audio clips)
Metrics: CER ROUGE-1 ROUGE-L BLEU WER latency
Baselines: GPT-3.5 GPT-4o Gemini-Pro T5-small (1 stage, w/o corrector) T5-base (1 stage) T5-small/T5-base 2-stage simple concatenation
Builds on: T5 Whisper VITS
Closest papers by meaning
- Regress, Don't Guess: A Regression-like Loss on Number Tokens for Language Models (ICML 2025 · similarity 0.86)
- MathPile: A Billion-Token-Scale Pretraining Corpus for Math (NeurIPS 2024 · similarity 0.85)
- A Compact Model for Mathematics Problem Representations Distilled from BERT (AAAI 2025 · similarity 0.85)
- OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset (NeurIPS 2024 · similarity 0.85)
- MASS: Mathematical Data Selection via Skill Graphs for Pretraining Large Language Models (ICML 2025 · similarity 0.85)
- MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.