Research map / Vision-language and multimodal / CLIP and zero-shot
SD-LoRA: Training-Time Structural Distillation into LoRA for Few-Shot Vision-Language Adaptation
Mohammed Rahman Sherif Khan Mohammad, Ardhendu Behera, Sandip Pradhan, Swagat Kumar, Yonghuai Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SD-LoRA, which adapts frozen CLIP-like vision-language models for few-shot tasks by distilling token-level structure into LoRA during training.
Problem
Few-shot adaptation of vision-language models must learn fine-grained visual distinctions from very limited supervision while preserving deployment efficiency; existing lightweight methods supervise only global image embeddings and underuse token-level information.
Approach
Training-time structural distillation that transfers token-level structure into LoRA weights, so no extra inference cost is added; the abstract indicates it targets the underuse of token-level supervision in global-embedding-based adapters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add training-time graph teacher with implicit distillation to CLIP-LoRA | CLIP-LoRA · 11 few-shot benchmarks, 16-shot | accuracy: 83.0 -> 85.0 | Improved |
| add graph teacher on top of LoRA+Cache (ablation) | LoRA + Cache · Caltech101 4-shot | accuracy: 93.0 -> 96.2 | Improved |
| add cache head on top of LoRA+Graph (ablation) | LoRA + Graph · EuroSAT 16-shot | accuracy: 93.1 -> 94.0 | Improved |
| structural training versus matched 336-trained cache baseline (ablation) | SD-LoRA · 4 datasets avg, 1-shot | accuracy: 71.3 -> 73.1 | Improved |
| higher training resolution (224 to 336) without structural supervision (ablation) | CLIP-LoRA+cache · 4 datasets avg, 1-shot | accuracy: 70.8 -> 71.3 | Improved |
| structural distillation on base-to-novel transfer | SD-LoRA · Base-to-novel avg 11, novel classes | accuracy: 80.73 -> 86.59 | Improved |
| structural distillation on base-class accuracy | SD-LoRA · Base-to-novel EuroSAT/Flowers/DTD base | accuracy: -0.25 EuroSAT, -1.22 Flowers, -1.16 DTD | Mixed |
| structural distillation on ImageNet OOD variants | SD-LoRA · ImageNet-A | accuracy: 49.63 vs MaPLe 50.90 | No clear effect |
| structural distillation on OOD average | SD-LoRA · ImageNet OOD avg | accuracy: 62.85 -> 63.26 | Improved |
| structural distillation effect on calibration | SD-LoRA · Cross-dataset transfer | ECE: 5.57 -> 2.65 | Improved |
| gains shrink with more shots | SD-LoRA · 11 few-shot benchmarks vs strongest baseline | accuracy: +2.4 (1-shot) to +0.4 (16-shot) | Mixed |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 2 on most shared results.
- PromptKD — PromptKD: Unsupervised Prompt Distillation for Vision-Language Models (CVPR 2024) · better on 5 of 5 shared results
- MaPLe — MaPLe: Multi-Modal Prompt Learning (CVPR 2023) · better on 0 of 1 shared result
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Experimental setup
Baselines: prompt tuning LoRA cache-based adapters
Builds on: LoRA CLIP
Closest papers by meaning
- LoRA Recycle: Unlocking Tuning-Free Few-Shot Adaptability in Visual Foundation Models by Recycling Pre-Tuned LoRAs (CVPR 2025 · similarity 0.90)
- Rethinking Few-Shot Adaptation of Vision-Language Models in Two Stages (CVPR 2025 · similarity 0.89)
- Hawaii: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Models (NeurIPS 2025 · similarity 0.89)
- A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models (CVPR 2024 · similarity 0.89)
- Exploring the Better Multimodal Synergy Strategy for Vision-Language Models (AAAI 2025 · similarity 0.89)
- LoRAtorio: An intrinsic approach to LoRA Skill Composition (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
