Research map / Large language models / Language, safety and interpretability
Natural Language Generation with Expert Standards
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
A doctoral consortium proposal outlining a dissertation on teaching and evaluating LLMs to follow expert-defined standards for NLG, summarizing prior STANDARDIZE and SPECIALEX work and planned generalization and expert-in-the-loop trust studies.
Problem
LLMs need to capture multiple constraints derived from expert-defined standards (education, language assessment, etc.) for quality-controlled text generation, and such standard-alignment needs to be evaluated with domain experts.
Approach
Treats expert standards as a specialized form of human preference and proposes retrieval-style in-context learning (STANDARDIZE) to convert CEFR and Common Core standards into knowledge artifacts that steer generation, plus a planned fine-tuning generalization study and an expert-in-the-loop trust score framework.
Key results
- STANDARDIZE yielded a 40% to 100% increase in precise accuracy for Llama2 and GPT-4, respectively, on standard-aligned narrative generation.
- SPECIALEX, a benchmark with 18 subtasks and 1,785 test instances, was introduced and used to evaluate 15 open and closed-source LLMs.
- Expert evaluations from language assessment specialists supported the ease of use and acceptability of STANDARDIZE outputs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use STANDARDIZE retrieval and in-context learning with knowledge artifacts extracted from standards | Llama2 · Open-ended narrative generation using CEFR and Common Core Standards | precise accuracy: 40% increase | Helped |
| Use STANDARDIZE retrieval and in-context learning with knowledge artifacts extracted from standards | GPT-4 · Open-ended narrative generation using CEFR and Common Core Standards | precise accuracy: 100% increase | Helped |
Limitations
- Remaining dissertation work (generalization study and trust framework) is proposed and not yet executed.
- Evaluation of the trust scoring framework with experts is planned but not reported.
Experimental setup
Datasets: CEFR Common Core Standards
Benchmarks: SPECIALEX
Metrics: precise accuracy
Builds on: STANDARDIZE SPECIALEX RLHF
Closest papers by meaning
- Neither Valid nor Reliable? Investigating the Use of LLMs as Judges (NeurIPS 2025 · similarity 0.87)
- Bridging the Knowledge Gap: Understanding User Expectations for Trustworthy LLM Standards (AAAI 2025 · similarity 0.86)
- LLMEval: A Preliminary Study on How to Evaluate Large Language Models (AAAI 2024 · similarity 0.86)
- Benchmarking LLMs' Judgments with No Gold Standard (ICLR 2025 · similarity 0.85)
- Measuring what Matters: Construct Validity in Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.85)
- On Evaluating LLM Alignment by Evaluating LLMs as Judges (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.