Research map / Large language models / AI and society
Symbol Description Reading
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Applies extractive and generative question-answering models in a four-step pipeline to read descriptions of mathematical symbols from scientific text, evaluated on the MFQuAD (MIDR) and Symlink (SemEval 2022 Task 12) datasets.
Problem
Symbol description reading: identifying the textual descriptions of mathematical identifiers in scientific documents, which is needed to embed formula semantics into LLMs and support downstream applications.
Approach
Treats symbol description reading as question answering ("What is the definition of <identifier>?") over passages retrieved around identifier occurrences, scoring spans with a reading comprehension model, and re-ranking candidate answers across occurrences with a non-neural classifier.
Key results
- Fine-tuned extractive QA (RoBERTa with LR re-ranking) reaches EM 0.671 and F1 0.716 on MFQuAD all identifiers, up from 0.416/0.450 without fine-tuning and above NPR's 0.477/0.529.
- Fine-tuned generative QA (unifiedqa-v2) reaches EM 0.655 and F1 0.694 on all MFQuAD identifiers and EM 0.224 on implicit identifiers, beating extractive QA (0.061) on implicit cases.
- On the Symlink task, the extractive QA approach achieves strict relation extraction F1 of 0.3279 versus 0.2698 for JBNU (about 21% improvement) and strict entity F1 of 0.4633 versus 0.4499.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MIDR-specific fine-tuning to the extractive QA pipeline (ablation) | Extractive QA · MFQuAD All | Exact Match: 0.416 -> 0.671 | Helped |
| Add MIDR-specific fine-tuning to the extractive QA pipeline (ablation) | Extractive QA · MFQuAD All | F1: 0.450 -> 0.716 | Helped |
| Use generative rather than extractive QA | MIDR · MFQuAD Implicit | Exact Match: 0.061 -> 0.224 | Helped |
| Use extractive rather than generative QA | MIDR · MFQuAD Missing | Exact Match: 0.744 -> 0.872 | Helped |
| Use QA instead of noun phrase ranking | MIDR · MFQuAD All | Exact Match: 0.477 -> 0.671 | Helped |
| Use extractive QA instead of JBNU | Symlink · Symlink test | Strict relation F1: 0.2698 -> 0.3279 | Helped |
| Use extractive QA instead of JBNU | Symlink · Symlink test | Strict entity F1: 0.4499 -> 0.4633 | Helped |
Surprising findings
- The generative T5 model struggles to recognize when a passage contains no description, performing worse on Missing identifiers than the simpler NPR baseline.
- Generative QA is much better than extractive QA on implicit descriptions, which require reading basic mathematical syntax.
Limitations
- Implicit identifier descriptions remain hard, with low absolute performance for extractive QA.
- Generative model fails to abstain when no description is present.
- Evaluation limited to two datasets and the MIDR results rely on 5-fold cross-validation over a single dataset.
Experimental setup
Datasets: SQuAD 2.0 MFQuAD Symlink (SemEval 2022 Task 12)
Benchmarks: MFQuAD Symlink (SemEval 2022 Task 12)
Metrics: Exact Match F1 entity F1 (strict, partial, type) relation precision/recall/F1
Baselines: NPR (Lynch et al. 2023) JBNU (Lee and Na 2022) Schubotz et al. 2017
Builds on: SQuAD 2.0 reading comprehension Schubotz et al. noun phrase ranking Kratzwald et al. answer re-ranking T5
Closest papers by meaning
- Exploring Equation as a Better Intermediate Meaning Representation for Numerical Reasoning of Large Language Models (AAAI 2024 · similarity 0.83)
- Mastering Symbolic Operations: Augmenting Language Models with Compiled Neural Networks (ICLR 2024 · similarity 0.83)
- A Compact Model for Mathematics Problem Representations Distilled from BERT (AAAI 2025 · similarity 0.83)
- OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text (ICLR 2024 · similarity 0.82)
- Large Language Models are Interpretable Learners (ICLR 2025 · similarity 0.82)
- GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (ICLR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.