Research map / Large language models / Language, safety and interpretability
Improving Natural Language Understanding for LLMs via Large-Scale Instruction Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds Hum, a 2.8M-instruction synthetic NLU corpus (IE, MRC, text classification, instruction generalist) with guideline, preference-rule, and format-variant synthesis, and fine-tunes six LLMs with LoRA to improve zero-shot NLU while largely preserving general capabilities.
Problem
Existing instruction data for LLMs focuses on information extraction and neglects machine reading comprehension, QA, and text classification, and its rigid formats cause overfitting and loss of general capabilities.
Approach
Synthesizes NLU instructions from many tasks and diversifies them via guideline synthesis (descriptions, examples, formats), GPT-4-generated preference rules that change labeling semantics, and format variants (JSON, text, markdown) to reduce overfitting.
Key results
- Hum fine-tuning improves average zero-shot NLU F1 across six LLMs by 3.1% (language understanding dimension), per the paper's summary.
- Hum-tuned Qwen2 reaches 69.41 average on the five NLU tasks with compound instructions, versus 49.57 for base Qwen2 and 57.27 for GPT-4.
- Hum-tuned Qwen2 average on the seven general dimensions is 67.00, versus 68.17 for base Qwen2; declines are reported for Qwen2 and Baichuan2 in general dimensions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Qwen2 on Hum | Qwen2 · Five NLU tasks (average) | F1: 64.15 -> 69.41 | Helped |
| Add examples and descriptions to basic-style instructions (ablation) | Qwen2 in-context instructions · FewRel | F1: 26.90 -> 45.07 | Helped |
| Add examples and descriptions to basic-style instructions (ablation) | Qwen2 in-context instructions · CrossNER | F1: 50.86 -> 58.14 | Helped |
| Add guideline synthesis (ablation) | Hum fine-tuning on Qwen2 · FewRel | F1: 41.20 -> 45.07 | Helped |
| Add guideline synthesis (ablation) | Hum fine-tuning on Qwen2 · CrossNER | F1: 60.50 -> 58.14 | Hurt |
| Add preference-rule synthesis (ablation) | Hum fine-tuning on Qwen2 · FewRel | F1: 44.36 -> 45.07 | Helped |
| Add preference-rule synthesis (ablation) | Hum fine-tuning on Qwen2 · CrossNER | F1: 57.90 -> 58.14 | No clear effect |
| Add format-variant synthesis (ablation) | Hum fine-tuning on Qwen2 · FewRel | F1: 37.11 -> 45.07 | Helped |
| Add format-variant synthesis (ablation) | Hum fine-tuning on Qwen2 · CrossNER | F1: 56.83 -> 58.14 | Helped |
| Mix basic and compound instructions instead of using only basic instructions (ablation) | Qwen2 fine-tuned on Hum · Five NLU tasks (average, compound) | F1: 65.48 -> 69.41 | Helped |
| Increase Hum training set from 10K to 2.8M samples (ablation) | Qwen2 fine-tuned on Hum · Five NLU tasks (average, compound) | F1: 63.14 -> 69.41 | Helped |
| Fine-tune Qwen2 on Hum | Qwen2 · General capabilities (average) | score: 68.17 -> 67.00 | Hurt |
Surprising findings
- Hum-tuned Qwen2 and Baichuan2 show a decline in average general-capability performance, and Hum-tuned coding performance drops for most models, which the authors attribute to JSON-format bias.
Limitations
- Declines in general capabilities for Qwen2 and Baichuan2 and in coding and multiple-choice knowledge tasks.
- Hum data focuses on factual extraction, leaving logical reasoning and multiple-choice formats less improved.
- Preference rule synthesis relies on GPT-4 generated rules and labels, and many training sources are self-built or reliant on distant supervision.
Experimental setup
Datasets: ACE2005 CoNLL2003 DuIE2.0 CMRC2018 ChnSentiCorp THUCNews InstructIE UniNER alpaca data COIG-CQIA
Benchmarks: CrossNER FewRel CCF Law C3 IMDB WSC XSum Lambada LCSTS RACE C-Eval CMMLU
Metrics: F1 accuracy ROUGE-1
Baselines: GPT-4 OneKE YAYI-UIE Llama3-iepile
Builds on: UIE InstructUIE IEPILE LlamaFactory
Closest papers by meaning
- MAmmoTH2: Scaling Instructions from the Web (NeurIPS 2024 · similarity 0.89)
- Enhancing NLU in Large Language Models Using Adversarial Noisy Instruction Tuning (AAAI 2025 · similarity 0.89)
- Evaluating the Zero-shot Robustness of Instruction-tuned Language Models (ICLR 2024 · similarity 0.88)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.88)
- MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks (ICLR 2026 · similarity 0.88)
- Evaluating Large Language Models at Evaluating Instruction Following (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.