Research map / Large language models / Language, safety and interpretability
Zero-Shot Learning for Materials Science Texts: Leveraging Duck Typing Principles
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MatDuck, a training-free zero-shot materials-science text mining method that generates ClassDefinition-style descriptors (inspired by duck typing) and reframes tasks as code-style in-context learning prompts for LLMs.
Problem
Zero-shot materials science text mining (NER, relation extraction, event argument extraction, paragraph/sentence classification, synthesis action retrieval, slot filling) where the same descriptor label has different meanings across tasks, making fine-tuned models hard to generalize.
Approach
Uses LLMs to generate multiple class definitions (with properties) for each descriptor, selects the definition best fitting the task texts by counting instances, and presents the descriptor and text as Python code for in-context classification, without any fine-tuning.
Key results
- MatDuck with Claude3.5 reaches 0.697 overall micro-F1 across tasks in zero-shot, above fine-tuned MatSciBERT (0.671) and Claude3.5 baseline (0.584).
- MatDuck with LLaMA3-8B backbone (0.629 macro-F1 overall) outperforms GPT-4 zero-shot (0.557), per the paper's reported comparisons.
- Paper reports an average zero-shot improvement of 11.3% across seven MSTM tasks; removing properties from class definitions drops Claude3.5 performance to 0.575.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply MatDuck to Claude3.5 | Claude3.5 · All tasks | macro-F1: 0.584 -> 0.697 | Helped |
| apply MatDuck to Claude3.5 | Claude3.5 · Synthesis Action Retrieval | macro-F1: 0.817 -> 0.874 | Helped |
| apply MatDuck to Claude3.5 | Claude3.5 · Slot Filling | macro-F1: 0.578 -> 0.554 | Hurt |
| add task-specific properties and annotations to descriptor class definitions (ablation) | MatDuck · All tasks | macro-F1: 0.575 -> 0.697 | Helped |
| add task-specific properties and annotations to descriptor class definitions (ablation) | MatDuck · All tasks | micro-F1: 0.508 -> 0.605 | Helped |
| use task adaptation to select descriptor class definitions (ablation) | MatDuck · All tasks | macro-F1: 0.613 -> 0.697 | Helped |
| use task adaptation to select descriptor class definitions (ablation) | MatDuck · All tasks | micro-F1: 0.541 -> 0.605 | Helped |
| use code-style ICL rather than natural-language ICL (ablation) | MatDuck · All tasks | macro-F1: 0.633 -> 0.697 | Helped |
| use code-style ICL rather than natural-language ICL (ablation) | MatDuck · All tasks | micro-F1: 0.556 -> 0.605 | Helped |
| vary the number of generated class-definition properties (ablation) | MatDuck · MSTM tasks | performance | Mixed |
Surprising findings
- Removing task-specific properties from class definitions drops performance below the original LLM baseline.
- Zero-shot MatDuck with Claude3.5 surpasses fine-tuned MatSciBERT on overall F1.
Limitations
- Relies on API-accessed closed LLMs for best results, with reproducibility dependent on model access.
- Performance drops with too many properties due to overly long contexts.
- Formulates tasks as classification, which may not cover generative extraction settings.
Experimental setup
Datasets: MatSci-NLP benchmark (eight materials science text datasets)
Benchmarks: MatSci-NLP
Metrics: micro-F1 macro-F1
Baselines: GPT-4 Claude3.5 GPT-3.5 LLaMA3-8B HoneyBee-7B HoneyBee-13B MatBERT MatSciBERT
Builds on: chain-of-thought prompting in-context learning HoneyBee MatSci-NLP
Closest papers by meaning
- MatExpert: Decomposing Materials Discovery By Mimicking Human Experts (ICLR 2025 · similarity 0.86)
- SciCode: A Research Coding Benchmark Curated by Scientists (NeurIPS 2024 · similarity 0.84)
- Construction and Application of Materials Knowledge Graph in Multidisciplinary Materials Science via Large Language Model (NeurIPS 2024 · similarity 0.84)
- Generative Hierarchical Materials Search (NeurIPS 2024 · similarity 0.84)
- Code-Generated Graph Representations Using Multiple LLM Agents for Material Properties Prediction (ICML 2025 · similarity 0.83)
- MarkushGrapher: Joint Visual and Textual Recognition of Markush Structures (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.