Research map / Large language models / Instruction and fine-tuning
Magical: Medical Lay Language Generation via Semantic Invariance and Layperson-tailored Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Magical, an asymmetric LoRA with a shared matrix A (semantic invariance via contrastive learning on probed layers) and multiple isolated B matrices switched by an external recommendation interface, for medical lay language generation on Cochrane, eLife, and PLOS Genetics.
Problem
Standard LoRA fine-tuned on multi-source heterogeneous medical lay language generation (MLLG) datasets fails to preserve semantic fidelity and to adapt to diverse lay styles.
Approach
An asymmetric LoRA: one shared down-projection A trained with a semantic contrastive loss on automatically identified semantically relevant layers, plus multiple isolated up-projections B, one per lay-style dataset, selected by a hard switch (recommendation-guided) rather than a learned router.
Key results
- On LLaMA3.1-8B, Magical reaches R-1 45.71 on Cochrane and 50.44 on eLife, versus 40.19 and 49.40 for LoRA, with 42M vs 62M trainable parameters.
- Magical has an average of 31.66% fewer trainable parameters than the second-best method while outperforming LoRA variants in most settings.
- Magical with a 100% accurate recommender still beats naive LoRA at 60% recommendation accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Magical instead of the strongest listed baseline | LLaMA3.1-8B-Instruct · Cochrane | R-1: 43.30 -> 45.71 | Helped |
| Use Magical instead of the strongest listed baseline | LLaMA3.1-8B-Instruct · eLife | BLEU: 8.71 -> 8.67 | Hurt |
| Add Semantic-Relevant Layer Identification (ablation) | Magical · Cochrane, eLife, and Plos_genetics | R-1: average 1.90%↓ | Helped |
| Add Semantic Contrastive Learning (ablation) | Magical · Cochrane, eLife, and Plos_genetics | BLEU: average 0.83%↓ | Helped |
| Use multiple B matrices instead of a single B matrix (ablation) | Magical · Cochrane, eLife, and Plos_genetics | R-1: average 4.46%↓ | Helped |
| Use switch-controlled selection instead of router-controlled selection (ablation) | Magical · Cochrane, eLife, and Plos_genetics | R-1: null | Helped |
| Set top-K layer selection to K=16 rather than K=8, 24, or 32 (ablation) | Magical · Cochrane | R-1: 44.74 -> 45.71; 45.11 -> 45.71; 44.02 -> 45.71 | Mixed |
| Use multiple smaller LoRAs instead of a single larger LoRA | LoRA on heterogeneous MLLG datasets · Cochrane, eLife, and Plos_genetics | R-1: null | Mixed |
| Decrease recommendation accuracy from 100% to 60% (ablation) | Magical · MLLG datasets | performance: null | Hurt |
| Use Magical instead of FFT | LLaMA3.1-8B-Instruct · Cochrane | R-1: 43.2 -> 45.71 | Helped |
Surprising findings
- Multiple smaller LoRAs, one per dataset, outperform a single larger LoRA trained jointly on heterogeneous datasets.
- A router-controlled selection of B matrices performs about as poorly as using a single B matrix.
Limitations
- The Recommendation Agent is not implemented; experiments use a manually specified B matrix (oracle 100% accuracy).
- Generalization to unseen lay styles is poor; only matrix A transfers reliably.
Experimental setup
Datasets: Cochrane eLife Plos_genetics EASI IN-Abs IN-Ext UK-Abs
Benchmarks: Cochrane eLife Plos_genetics EASI
Metrics: ROUGE-1 ROUGE-2 ROUGE-L BLEU BERTScore
Baselines: Prompt LoRA rsLoRA DoRA PiSSA HydraLoRA FFT o3-mini
Builds on: LoRA HydraLoRA rsLoRA DoRA PiSSA
Closest papers by meaning
- Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Experts (ICLR 2025 · similarity 0.85)
- HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation (ICML 2025 · similarity 0.84)
- Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation (NeurIPS 2026 · similarity 0.83)
- Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-World Multi-Turn Dialogue (AAAI 2024 · similarity 0.83)
- PediatricsGPT: Large Language Models as Chinese Medical Assistants for Pediatric Applications (NeurIPS 2024 · similarity 0.83)
- CareBot: A Pioneering Full-Process Open-Source Medical Language Model (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.