Research map / Architectures and efficiency / Attention and transformers
Towards a Transformer-Based Reverse Dictionary Model for Quality Estimation of Definitions (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compares DistilBERT, DistilGPT-2 and XLNet fine-tuned for the reverse dictionary task on Hill et al. data, and applies them to score dictionary quality in a serious game called The Dictionary Game.
Problem
Reverse dictionary (guessing a word from its definition) and using it to estimate the quality of dictionaries generated through a word-definition game.
Approach
Fine-tunes pretrained transformers (DistilBERT, DistilGPT-2, XLNet) with a pooled-representation softmax classifier over words, plus an auxiliary POS-prediction task, and uses rank-based metrics of the fine-tuned models to quantify dictionary quality.
Key results
- XLNet on the description test set reports median rank 3 and top-1/10/100 of .38/.65/.85, slightly improving top-1 over prior state of the art the authors know of.
- XLNet is reported to generalize best, followed by DistilBERT, then DistilGPT-2.
- Degree of certainty is found inadequate for dictionary quality: DistilBERT gets certainty 0.55 on Merriam-Webster while averaging median rank 0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use XLNet instead of DistilBERT or DistilGPT-2 | Reverse dictionary model · Description test set | overall task generalization | Helped |
| Use DistilBERT instead of the other compared models | Dictionary quality evaluation · Dictionaries | average median rank | Helped |
| Use the model's degree of certainty as the dictionary quality measure | DistilBERT dictionary evaluation · Merriam-Webster dictionary | degree of certainty as a quality measure: 0.55; 0 | Hurt |
Surprising findings
- Degree of certainty can be high for synonym-heavy dictionaries even when the rank-based quality is poor, making it an unreliable quality measure.
- A dictionary with fewer than 24 words can achieve a better average rank than one with more than 110 words.
Limitations
- Short definitions in the dictionaries may favor bidirectional models, confounding comparisons.
- Average rank is highly variable for small dictionaries, so dictionary quality measures are noisy.
- Only three transformer variants and a single dataset are evaluated.
Experimental setup
Datasets: Hill et al. dictionary word-definition data Merriam-Webster dictionary definitions
Benchmarks: Hill et al. reverse dictionary description test set
Metrics: median rank top-1/10/100 accuracy rank variance degree of certainty
Baselines: Multi-Channel Reverse Dictionary Model previous state-of-the-art reverse dictionary models
Builds on: BERT XLNet DistilBERT GPT-2
Closest papers by meaning
- Re-evaluating Open-ended Evaluation of Large Language Models (ICLR 2025 · similarity 0.79)
- Adventures of Trustworthy Vision-Language Models: A Survey (AAAI 2024 · similarity 0.79)
- Understanding Transformers via N-Gram Statistics (NeurIPS 2024 · similarity 0.79)
- Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Models (NeurIPS 2024 · similarity 0.79)
- TRANSFORMER EXPLAINER: Interactive Learning of Text-Generative Models (AAAI 2025 · similarity 0.78)
- Do Efficient Transformers Really Save Computation? (ICML 2024 · similarity 0.78)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.