Research map / Image generation / Fast diffusion sampling
Few-shot Learner Parameterization by Diffusion Time-steps
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TiF learner, which trains class-specific LoRA adapters on a text-conditioned Stable Diffusion and classifies images by time-step-weighted denoising reconstruction error, using diffusion time-steps to separate nuanced class attributes from spurious backgrounds in few-shot learning.
Problem
Few-shot classification of fine-grained or customized categories with foundation models suffers from spurious correlations, since discriminative adapters latch onto visually prominent background attributes instead of nuanced class attributes.
Approach
Nuanced class attributes are lost at earlier diffusion time-steps than visually prominent spurious attributes, so per-class LoRA-adapted diffusion denoisers evaluated at small, hyper-parameter-free time-step weights isolate class-defining attributes for de-biased classification.
Key results
- TiF learner reaches 48.5% on 1-shot FGVCAircraft, a +24.2 gain over its zero-shot diffusion base (24.3%).
- On DukeMTMC-reID 16-shot, TiF learner w/o c reaches 91.6% rank-1 versus 82.6% for Tip-Adapter-F, a gain of about 9 points.
- Reported gains over the best baseline reach 13.7% on FGVCAircraft, 21.6% on DukeMTMC-reID and 16% on VeRi-776.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add class names to the prompt (ablation) | TiF learner · FGVCAircraft, 1-shot test | accuracy: 40.4 -> 48.5 | Helped |
| Add class names to the prompt (ablation) | TiF learner · ISIC2019, 16-shot test | macro F1: 43.8 -> 44.7 | Helped |
| Expand LoRA injection from the last block to include w1 and w2 (ablation) | TiF learner · FGVCAircraft and DukeMTMC-reID, 16-shot | accuracy / rank-1 accuracy: 78.1 -> 79.9; 91.3 -> 91.6 | Mixed |
| Replace the proposed inference weight with the ELB weight (ablation) | TiF learner · FGVCAircraft, 16-shot test | accuracy: 79.9 -> 71.2 | Hurt |
| Replace the proposed inference weight with the ELB weight (ablation) | TiF learner · DukeMTMC-reID, 16-shot query | rank-1 accuracy: 91.6 -> 81.8 | Hurt |
| Replace the proposed inference weight with the PDAE weight (ablation) | TiF learner · FGVCAircraft and DukeMTMC-reID, 16-shot | accuracy / rank-1 accuracy: 79.9 -> 78.1; 91.6 -> 90.8 | Mixed |
| Use TiF learner instead of Tip-Adapter-F | OpenCLIP ViT-H/14 few-shot classification · FGVCAircraft, 16-shot test | accuracy: 67.4 -> 79.9 | Helped |
| Use TiF learner instead of Tip-Adapter-F | OpenCLIP ViT-H/14 few-shot classification · ISIC2019, 16-shot test | macro F1: 40.3 -> 44.7 | Helped |
| Use TiF learner instead of MaPLe | few-shot re-identification · DukeMTMC-reID, 16-shot query | rank-1 accuracy: 63.0 -> 91.6 | Helped |
| Use TiF learner instead of Tip-Adapter-F | OpenCLIP ViT-H/14 few-shot re-identification · VeRi-776, 16-shot query | rank-1 accuracy: 85.4 -> 96.8 | Helped |
Surprising findings
- Zero-shot prompt tuning methods (CoOp, Co-CoOp, MaPLe) perform poorly on reID datasets lacking class names, suggesting prompts alone cannot describe fine-grained classes.
Limitations
- Does not handle hierarchical or coarse-grained class attributes, which the authors say is out of scope.
- Fine-grained spurious attributes correlated with the class can still be isolated at small time-steps and are not removed without extra prior.
- Low-shot gains are smaller on some tasks and depend on the discriminative ability of the zero-shot base model.
Experimental setup
Datasets: FGVCAircraft ISIC2019 DukeMTMC-reID VeRi-776
Benchmarks: FGVCAircraft ISIC2019 DukeMTMC-reID VeRi-776
Metrics: accuracy macro F1 rank-1 accuracy
Baselines: CLIP zero-shot CoOp Co-CoOp MaPLe OpenCLIP linear-probe Tip-Adapter Tip-Adapter-F CaFo CLIP-Adapter Diffusion Classifier
Builds on: Diffusion Classifier Tip-Adapter DreamBooth LoRA CoOp
Closest papers by meaning
- Few-Shot Diffusion Models Escape the Curse of Dimensionality (NeurIPS 2024 · similarity 0.88)
- Random Conditioning for Diffusion Model Compression with Distillation (CVPR 2025 · similarity 0.87)
- Leveraging Normalization Layer in Adapters with Progressive Learning and Adaptive Distillation for Cross-Domain Few-Shot Learning (AAAI 2024 · similarity 0.87)
- On improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Models (NeurIPS 2024 · similarity 0.87)
- Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation (NeurIPS 2024 · similarity 0.86)
- BECLR: Batch Enhanced Contrastive Few-Shot Learning (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.