Research map / Large language models / Language, safety and interpretability
Tag-LLM: Repurposing General-Purpose LLMs for Specialized Domains
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Tag-LLM appends learned continuous domain and function tags to a frozen-weight LLM's embedding layer, trained in a three-stage protocol, to repurpose LLaMA-7B for protein, chemistry, and drug-discovery tasks and multilingual translation.
Problem
General-purpose LLMs perform poorly in specialized domains underrepresented in pretraining, such as protein sequences, SMILES molecules, and drug-target interactions, while full fine-tuning is costly and catastrophic forgetting harms reasoning ability.
Approach
Learn reusable input tags as continuous embeddings: domain tags (trained by next-token prediction on unlabeled in-domain data) delimit specialized inputs, and function tags (trained on labeled data, with optional task-specific regression heads) encode tasks, so tags can be composed for zero-shot generalization.
Key results
- Tag-LLaMA reaches MSE 0.005 on descriptor and 0.008 on QED, beating the best specialized expert (0.012 and 0.04) and all PEFT baselines.
- On TDC DrugComb, Tag-LLaMA achieves MAE 12.21 vs. 16.85 for the best expert MLP, a state-of-the-art result on that task.
- On BindingDB_Patent, Tag-LLaMA gets Pearson r 0.527, ranking third on the leaderboard behind Otter-Knowledge (0.588) and PLM (0.538).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Tag-LLM instead of the strongest reported baseline | LLaMA-7B · Descriptor | MSE: 0.012 -> 0.005 | Helped |
| use Tag-LLM instead of the strongest reported baseline | LLaMA-7B · QED | MSE: 0.016 -> 0.008 | Helped |
| use Tag-LLM instead of the strongest reported baseline | LLaMA-7B · Drug Combination | MAE: 16.85 -> 12.21 | Helped |
| use Tag-LLM instead of the strongest reported baseline | LLaMA-7B · Binding Affinity | Pearson r: 0.588 -> 0.527 | Hurt |
| add domain tags (ablation) | Tag-LLM · Drug Combination | MAE: 12.39 -> 12.21 | Helped |
| add function tags (ablation) | Tag-LLM · Drug Combination | MAE: 21.14 -> 12.21 | Helped |
| add a regression head instead of generating digits with the language modeling head (ablation) | Tag-LLM · Drug Combination | MAE: 23.42 -> 12.21 | Helped |
| enrich the domain tag with task knowledge (ablation) | Tag-LLM · Drug Combination | MAE: 12.21 -> 12.10 | Helped |
| increase the DC tag length (ablation) | Tag-LLM · Drug Combination | test error | Mixed |
| add a regression head (ablation) | PEFT baselines · Descriptor, QED, Drug Combination, Binding Affinity | task metric | Helped |
| replace learned domain tag embeddings with text-form domain info (ablation) | Prompt Tuning and LoRA · specialized scientific tasks | task metric | No clear effect |
Surprising findings
- Simply writing the domain name as text tokens yields only marginal gains over prompt tuning and LoRA, suggesting learned domain tags carry information text cannot.
- Removing function tags hurts DC performance far more (MAE 21.14) than removing domain tags (MAE 12.39).
Limitations
- Only regression tasks are explored; classification and structured prediction are left to future work.
- Binding affinity performance trails specialized ensemble and PLM models and LoRA on BA-style correlation.
- LLaMA is mainly English-trained, so translation into non-English targets underperforms.
Experimental setup
Datasets: OPUS-100 FLORES-101 binding_affinity (jglaser) DrugComb_CSS (TDC) BindingDB_Patent (TDC)
Benchmarks: FLORES-101 Descriptor (peptides) QED (RDKit) TDC DrugComb_CSS TDC BindingDB_Patent
Metrics: spBLEU MSE MAE Pearson r ROUGE COMET
Baselines: LoRA Prompt Tuning Linear Probing GPT-4 Galactica LlaSMol Text+Chem T5 M2M XGLM BLOOM GPT-3 Otter-Knowledge
Builds on: prompt tuning prefix tuning LoRA
Closest papers by meaning
- LICO: Large Language Models for In-Context Molecular Optimization (ICLR 2025 · similarity 0.87)
- SciLitLLM: How to Adapt LLMs for Scientific Literature Understanding (ICLR 2025 · similarity 0.87)
- Omni-Mol: Multitask Molecular Model for Any-to-any Modalities (NeurIPS 2025 · similarity 0.87)
- Training a Scientific Reasoning Model for Chemistry (NeurIPS 2025 · similarity 0.87)
- CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets (ICLR 2024 · similarity 0.86)
- LLM Augmented LLMs: Expanding Capabilities through Composition (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.