Research map / Large language models / Instruction and fine-tuning
Efficient Transcoder Adaptation for Fine-Tuned Models: Revealing Medical Reasoning Mechanisms in Large Language Models
Zhouxing Tan, Hanlin Xue, Yulong Wan, Ruochong Xiong, Xu Chu, Xiang Li, Junfei Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune only MLP layers with frozen attention (FTFA) instead of full-parameter SFT | Gemma-2-2b · Medical diagnosis benchmark | accuracy: 50.06 -> 52.27 | Improved |
| fine-tune only MLP layers with frozen attention (FTFA) instead of full-parameter SFT | Gemma-2-2b · Medical diagnosis benchmark | BERTScore: 54.18 -> 56.36 | Improved |
| apply FTFA fine-tuning relative to the untuned base model | Gemma-2-2b · Medical diagnosis benchmark | accuracy: 50.79 -> 52.27 | Improved |
| use base-model pretrained transcoders directly on the fine-tuned model instead of PAT adaptation (ablation) | FTFA Gemma2-2b replacement model · Top-1 accuracy of replacement model | Top-1 accuracy: nearly zero Top-1 accuracy | Hurt |
| adapt transcoders with PAT instead of training from scratch | Gemma2-2b transcoders (16k) · Transcoder training cost | training tokens: 1B -> 1M | Improved |
| adapt transcoders with PAT instead of training from scratch | Gemma2-2b transcoders (16k) · Transcoder training cost | H100 hours: 210 -> 0.2 | Improved |
Closest papers by meaning
- TAIA: Large Language Models are Out-of-Distribution Data Learners (NeurIPS 2024 · similarity 0.86)
- OctoMed: Data Recipes for State-of-the-Art Multimodal Medical Reasoning (CVPR 2026 · similarity 0.86)
- Diagnosing Transformers: Illuminating Feature Spaces for Clinical Decision-Making (ICLR 2024 · similarity 0.86)
- HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation (ICML 2025 · similarity 0.86)
- Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation (NeurIPS 2026 · similarity 0.85)
- Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Experts (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
