Research map / Large language models / Retrieval and knowledge
NewTerm: Benchmarking Real-Time New Terms for Large Language Models with Annual Updates
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NewTerm, a highly automated, annually updatable benchmark that tests LLMs' understanding of new dictionary terms (new words, new phrases, old words with new meanings) across three tasks (COMA, COST, CSJ), showing over 20% accuracy drops on new terms.
Problem
LLMs have knowledge cutoffs and struggle with real-time new terms, but existing benchmarks focus on outdated facts or rely heavily on human effort, making continual updates costly; new terms remain largely unexplored.
Approach
Automatically build multiple-choice and judgment questions for newly added dictionary terms using GPT-4 with separated generation of correct and incorrect choices (via related-term generation: synonyms, antonyms, meaning guessing, partial synonyms), LLM filtering, and perplexity-based selection, so the benchmark can be refreshed annually with minimal human effort.
Key results
- Base vs Gold settings show an average 25.63-point accuracy drop caused by new terms across evaluated LLMs (Gold avg 87.11 vs Base avg 57.65 on NewTerm 2022).
- Human filtering agreed with automatic answers in 82.41% of cases, with overall accuracy of 81.06%, and Fleiss' Kappa of 0.70.
- Learned new terms rise by +10.70 accuracy after knowledge-cutoff updates versus +5.37 for unlearned terms; there are about 50% more learned terms in 2022 than in 2023.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Provide the new term's meaning in the prompt (Gold rather than Base) | LLMs evaluated on NewTerm · NewTerm 2022 and 2023 | accuracy: Base - Gold: -25.63 on average | Helped |
| Update the knowledge cutoff to a newer model version | Claude and GPT model series · NewTerm 2022 | accuracy (Base): +3.57 | Helped |
| Update the knowledge cutoff to a newer model version | Claude and GPT model series · NewTerm 2023 | accuracy (Base): +0.10 | Helped |
| Use a larger model within the same model series | LLM series evaluated on NewTerm · NewTerm | accuracy (Gold): +9.39 | Helped |
| Use a larger model within the same model series | LLM series evaluated on NewTerm · NewTerm | Base - Gold accuracy gap: -25.13 vs -26.94 for smaller and larger LLMs, respectively | Hurt |
| Evaluate on NewTerm 2023 rather than NewTerm 2022 | LLMs evaluated under Gold · NewTerm 2022 vs NewTerm 2023 | accuracy: +0.85 on average | No clear effect |
| Evaluate on NewTerm 2023 rather than NewTerm 2022 | LLMs evaluated under Base · NewTerm 2022 vs NewTerm 2023 | accuracy: +8.51 on average | Helped |
| Move from unlearned terms to learned terms after a knowledge-cutoff update | LLMs evaluated on NewTerm 2022 · NewTerm 2022 | accuracy improvement after update: +10.70 vs +5.37 for learned and unlearned terms, respectively | Helped |
| Use few-shot rather than zero-shot prompting | LLMs evaluated on NewTerm · NewTerm | accuracy | No clear effect |
Surprising findings
- Larger LLMs achieve higher Gold accuracy but show no reduction in the new-term performance drop (Base-Gold gap grows slightly for larger models).
- Knowledge-cutoff updates cover only some new terms and do not generalize to more distant new terms; overlap of learned terms across model series is limited.
Limitations
- Relies on high-performance LLMs for generation and filtering; errors propagate across cascaded subprocesses (about 81% accuracy after human filtering).
- Coverage limited to 300 new terms per year from English online dictionaries, which may introduce bias.
- Hard to control variables across years because the collected new terms differ in number and distribution.
Experimental setup
Datasets: online dictionary update logs (Cambridge, Collins, Oxford)
Benchmarks: NewTerm 2022 NewTerm 2023
Metrics: accuracy
Baselines: GPT-4-0613 GPT-4-1106-preview GPT-4-0125-preview GPT-3.5-turbo-0613 GPT-3.5-turbo-0125 Claude-3-opus Llama-3-Instruct-70B
Builds on: PromptSource Sentence-BERT Phrase-BERT
Closest papers by meaning
- BenchmarkCards: Standardized Documentation for Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.90)
- tinyBenchmarks: evaluating LLMs with fewer examples (ICML 2024 · similarity 0.89)
- Automating Dataset Updates Towards Reliable and Timely Evaluation of Large Language Models (NeurIPS 2024 · similarity 0.88)
- Measuring what Matters: Construct Validity in Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.87)
- SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research (AAAI 2024 · similarity 0.87)
- Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.