Research map / Large language models / Language, safety and interpretability
A Compact Model for Mathematics Problem Representations Distilled from BERT
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills a 3-layer, 312-hidden compact BERT-style model for math word problem representations using POS/MLM pretraining, bottleneck-based hidden-state and GTS-logit distillation, and weight averaging.
Problem
Task-agnostic compact models distilled from BERT lose math-related knowledge and perform poorly on math word problem (MWP) solving, while LLM encoders are too large and slow for resource-constrained math solvers.
Approach
Task-specific knowledge distillation from a fine-tuned Chinese BERT teacher into a 3-layer student, combining MWP-domain pretraining (MLM plus POS tagging), uniform layer-mapped hidden-state distillation through bottleneck linear layers, GTS-decoder soft/hard-label prediction distillation, and parameter averaging to avoid forgetting.
Key results
- The 3-layer, 312-hidden student (10.3M params) reaches 80.23% on Math23k and 80.46% on Ape-clean, versus 84.07% and 84.33% for the BERT teacher (about 95% of teacher performance on average).
- Compared with TinyBERT (L=4, H=312, 14.5M params, average 64.12%), DistilBERT (67M, average 68.92%) and MiniLM (66M, average 67.27%) under the same GTS decoder, the proposed model averages 80.35%.
- Ablation: removing pretraining drops Math23k accuracy from 80.23% to 77.5%, and removing hidden-state distillation drops it to 76%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add pretraining (ablation) | Our work · Math23k | accuracy | Helped |
| Add pretraining (ablation) | Our work · Ape-clean | accuracy | Helped |
| Add POS tagging to pretraining (ablation) | Our work · Math23k | accuracy | Helped |
| Add POS tagging to pretraining (ablation) | Our work · Ape-clean | accuracy | Helped |
| Add MLM to pretraining (ablation) | Our work · Math23k | accuracy | Helped |
| Add MLM to pretraining (ablation) | Our work · Ape-clean | accuracy | Helped |
| Add hidden-state distillation (ablation) | Our work · Math23k | accuracy | Helped |
| Add hidden-state distillation (ablation) | Our work · Ape-clean | accuracy | Helped |
| Add prediction distillation (ablation) | Our work · Math23k | accuracy | Helped |
| Add prediction distillation (ablation) | Our work · Ape-clean | accuracy | Helped |
| Add soft labels to prediction distillation (ablation) | Our work · Math23k | accuracy | Helped |
| Add soft labels to prediction distillation (ablation) | Our work · Ape-clean | accuracy | Helped |
Surprising findings
- Fine-tuning BERT on Math23k did not change its POS tagging F1 (79.51% before and after), suggesting POS knowledge is retained during task fine-tuning.
- Relation-based distillation (attention, FSP, NST matrices) performed worse than hidden-state distillation for MWP solving.
Limitations
- Evaluated only on Chinese MWP datasets (Math23k, Ape-clean) with a single GTS decoder.
- Does not use existing task-agnostic compact models as initialization; the authors note this as future work.
- Teacher is a fine-tuned BERT, and the paper leaves a stronger math-specific teacher and larger math corpora to future work.
Experimental setup
Datasets: Math23k Ape210k Ape-clean HMWP CM17k
Benchmarks: Math23k Ape-clean
Metrics: equation accuracy answer accuracy average accuracy F1 (POS tagging probe)
Baselines: DistilBERT MiniLM TinyBERT BERT (teacher)
Builds on: TinyBERT DistilBERT MiniLM GTS Model soups (weight averaging)
Closest papers by meaning
- Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages (ICLR 2025 · similarity 0.88)
- Learning Theorem Rationale for Improving the Mathematical Reasoning Capability of Large Language Models (AAAI 2025 · similarity 0.88)
- Token-Supervised Value Models for Enhancing Mathematical Problem-Solving Capabilities of Large Language Models (ICLR 2025 · similarity 0.87)
- WASD: Wasserstein-based Knowledge Distillation for Large Language Models (NeurIPS 2026 · similarity 0.87)
- Neuro-Symbolic Data Generation for Math Reasoning (NeurIPS 2024 · similarity 0.87)
- MathFimer: Enhancing Mathematical Reasoning by Expanding Reasoning Steps through Fill-in-the-Middle Task (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.