Research map / Large language models / Agents, code and math
IDGen: Item Discrimination Induced Prompt Generation for LLM Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IDGen, an item-discrimination-guided framework that generalizes seed prompts via "instruction gradient" and "response gradient" strategies with self-correction, and trains discrimination and difficulty estimators (Baichuan2-13B) to evaluate LLMs.
Problem
Static LLM evaluation sets lose discriminative power as models improve; existing instruction datasets and benchmarks do not systematically measure how well items separate high- and low-performing models.
Approach
Adapt Item Discrimination theory from educational assessment to score prompts by how well they separate LLMs, use this to guide prompt generalization (rule-based strategies plus response-conditioned question generation), add a CoT-based self-correction check for math items, and train Baichuan2-13B-based estimators for discrimination and difficulty.
Key results
- On expert-scored evaluation of five models, the IDGen hard-seed dataset gives the lowest mean score (51.92) and highest variance (10.06), versus about 67-71 mean and variance below 3.2 for SELF-INSTRUCT and WizardLM baselines.
- Generalization questions reach 94.0% usability for general text and 96.4% for math under human annotation, with discrimination index rising from 0.08 to 0.17 (general) and 0.09 to 0.20 (math) relative to seed data.
- The discrimination estimation model agrees with human evaluation at 0.72, and the difficulty model reaches 0.70 consistency with gold labels versus 0.52 for humans.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply IDGen to more challenging seed questions | LLM evaluation datasets · Public datasets | Discrimination Indexes: 0.140 -> 0.204 | Helped |
| Apply IDGen to more challenging seed questions | LLM evaluation datasets · Public datasets | Difficulty Score: 1.319 -> 1.941 | Helped |
| Generalize SELF-INSTRUCT seed data with IDGen | SELF-INSTRUCT seed data · Public datasets | Difficulty Score: 1.319 -> 1.541 | Helped |
| Generalize seed questions with IDGen | Seed questions · General text questions | Discrimination Indexes: 0.08 -> 0.17 | Helped |
| Generalize seed questions with IDGen | Seed questions · Mathematical questions | Discrimination Indexes: 0.09 -> 0.20 | Helped |
| Use challenging seed questions for IDGen | Five-model evaluation dataset · Five-model evaluation | Mean: 67.03 -> 51.92 | Hurt |
| Use challenging seed questions for IDGen | Five-model evaluation dataset · Five-model evaluation | Var.: 3.20 -> 10.06 | Helped |
| Use GPT-4-turbo instead of Hunyuan for generation (ablation) | IDGen · Small seed-data generation | High discrimination items: 5 -> 14 | Helped |
| Use Qwen instead of Hunyuan for generation (ablation) | IDGen · Small seed-data generation | High discrimination items: 5 -> 3 | Hurt |
| Use both Hunyuan-standard and Hunyuan-pro to judge question usability (ablation) | Mathematical question CoT check · Mathematical question usability | Generalization Data Usability: 87.0\% -> 90.0\% | Helped |
| Use both Hunyuan-standard and Hunyuan-pro to judge and correct questions (ablation) | Mathematical question CoT check · Mathematical question usability | Correction Data Usability: 93.3\% -> 96.4\% | Helped |
Surprising findings
- The difficulty estimation model agreed more with gold difficulty labels (0.70) than human annotators did (0.52), suggesting models may find human-hard items simple.
- Higher mean difficulty did not always imply higher discrimination; the paper notes items all answered wrongly are difficult but non-discriminative.
Limitations
- Framework depends on the strength of the generator LLMs (Hunyuan), with weaker generators giving lower-quality data.
- Reference answers for math problems requiring strong logical reasoning are not reliably accurate.
- Evaluation relies on a small number of experts and on Hunyuan-based judging, and the comparison uses a different scoring scale and evaluated model set than prior datasets.
Experimental setup
Datasets: Self-Instruct seed data (175 instructions) Alpaca (52k) Instruction Tuning with GPT-4 dataset WizardLM dataset Self-Instruct dataset In-house 6,000-instance seed set (Chinese and English)
Benchmarks: IDGen generated prompt set (general text and math)
Metrics: mean score variance across models discrimination index difficulty score usability rate agreement with human labels
Baselines: SELF-INSTRUCT Instruction Tuning with GPT-4 WizardLM
Builds on: Self-Instruct Evol-Instruct Item Discrimination theory Chain-of-thought prompting
Closest papers by meaning
- Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks (ICML 2026 · similarity 0.88)
- Efficient multi-prompt evaluation of LLMs (NeurIPS 2024 · similarity 0.87)
- Evaluating Large Language Models at Evaluating Instruction Following (ICLR 2024 · similarity 0.86)
- From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set (ICML 2025 · similarity 0.86)
- SELF-[IN]CORRECT: LLMs Struggle with Discriminating Self-Generated Responses (AAAI 2025 · similarity 0.86)
- Evaluating the Evaluator: Measuring LLMs’ Adherence to Task Evaluation Instructions (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.