Research map / Large language models / Instruction and fine-tuning
The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune on example-level instructions instead of leaving as base model (Cohere-command vs Cohere-base, 52B) (ablation) | Cohere 52B · Implicature resolution, 5-shot | accuracy: 65.1 -> 75.4 | Helped |
| example-level instruction tuning (best Example IT model vs best base model GPT-3) | LLMs · Implicature resolution, 5-shot | accuracy: 68.7 -> 82.0 | Helped |
| benchmark-level instruction tuning instead of base models | T0-11B, Flan-T5-11B · Implicature resolution, 5-shot | accuracy: 68.7 -> 61.7 | Hurt |
| fine-tune on dialogue data (BlenderBot) | BlenderBot-2.7B · Implicature resolution, 5-shot | accuracy: 53.3 | Hurt |
| add 5-shot chain-of-thought prompting | GPT-4 · Implicature resolution | accuracy: 82.0 -> 86.5 | Helped |
| add 5-shot chain-of-thought prompting | text-davinci-001 · Implicature resolution | accuracy: 74.5 -> 67.3 | Hurt |
| add 5-shot chain-of-thought prompting | Cohere-command-52b · Implicature resolution | accuracy: 75.4 -> 75.3 | No clear effect |
| add 5-shot chain-of-thought prompting | text-davinci-003 · Implicature resolution | accuracy: 79.7 -> 83.6 | Helped |
| add few-shot examples (k=0 to 5) | base models · Implicature resolution | accuracy: 57.7 -> 68.7 (GPT-3-175B) | Helped |
| add few-shot examples (k=0 to 5) | Benchmark IT models · Implicature resolution | accuracy: 55.6 -> 47.0 (T0-11B) | Hurt |
| scale model size from Ada to Davinci | text-<engine>-001 · Implicature resolution, 5-shot | accuracy: 57.6 -> 74.5 | Helped |
| use random labels in few-shot examples | Cohere-command-52B and text-davinci-001 · Implicature resolution | accuracy | No clear effect |
Closest papers by meaning
- Uncovering Gaps in How Humans and LLMs Interpret Subjective Language (ICLR 2025 · similarity 0.87)
- Potemkin Understanding in Large Language Models (ICML 2025 · similarity 0.87)
- Failing to Falsify: Evaluating and Mitigating Confirmation Bias in Language Models (NeurIPS 2026 · similarity 0.86)
- Sparsity-Guided Holistic Explanation for LLMs with Interpretable Inference-Time Intervention (AAAI 2024 · similarity 0.86)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.86)
- Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.