Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
MOKD: Cross-domain Finetuning for Few-shot Classification via Maximizing Optimized Kernel Dependence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MOKD, a bi-level HSIC-based finetuning method for cross-domain few-shot classification that optimizes the Gaussian kernel bandwidth for test power before maximizing label dependence and minimizing sample dependence.
Problem
Nearest centroid classifier (NCC) representations in cross-domain few-shot classification show high similarity between samples from different classes, causing misclassification and overfitting on scarce labeled support data.
Approach
Shows NCC loss lower-bounds an HSIC-based objective, then optimizes the kernel bandwidth to maximize HSIC test power (HSIC/sqrt(variance)) and trains a linear head to maximize label-representation HSIC while minimizing representation-representation HSIC.
Key results
- Trained on ImageNet only, MOKD averages 67.3% over all datasets vs URL's 65.9% and achieves average rank 1.4; best on 10 of 13 datasets.
- Trained on all datasets, MOKD averages 76.3% over all datasets vs URL's 75.8%, with average rank 2.2.
- Ablation shows removing HSIC(Z,Z) causes significant performance drop attributed to overfitting; Gaussian kernel outperforms IMQ kernel on most datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MOKD instead of URL | URL few-shot adaptation pipeline · Meta-Dataset, train on ImageNet only, Average Unseen | accuracy: 66.6 -> 68.1 | Helped |
| use MOKD instead of URL | URL few-shot adaptation pipeline · Meta-Dataset, train on All Datasets, Average Unseen | accuracy: 69.4 -> 70.3 | Helped |
| add test power maximization (ablation) | MOKD · Meta-Dataset | accuracy | Helped |
| add the HSIC(Z, Z) regularization term (ablation) | MOKD · Meta-Dataset | accuracy | Helped |
| replace the Gaussian kernel with IMQ (ablation) | MOKD · Meta-Dataset, train on All Datasets, Textures | accuracy: 76.5$\pm$0.7 -> 76.2$\pm$0.7 | Hurt |
| replace the Gaussian kernel with IMQ (ablation) | MOKD · Meta-Dataset, train on All Datasets, Birds | accuracy: 80.4$\pm$0.8 -> 80.5$\pm$0.8 | No clear effect |
| replace the Gaussian kernel with IMQ (ablation) | MOKD · Meta-Dataset, train on All Datasets, CIFAR-100 | accuracy: 63.9$\pm$1.0 -> 63.7$\pm$1.0 | Hurt |
| increase gamma (ablation) | MOKD · ImageNet | accuracy | Helped |
| increase gamma (ablation) | MOKD · MNIST | accuracy | Hurt |
Surprising findings
- Removing the HSIC(Z,Z) regularization term causes a significant accuracy drop, attributed to overfitting.
- MOKD is sometimes faster than URL (e.g., Omniglot, Fungi) despite being bi-level, because it avoids repeated prototype computation.
Limitations
- Bandwidth selected by grid search rather than gradient-based optimization, acting as a stop-gradient variant of the bi-level problem
- Fails to outperform URT on Quick Draw and URL-like baselines on Fungi (likely overfitting)
- Time consumption varies with support set size, so efficiency is not uniformly better
Experimental setup
Datasets: Meta-Dataset ImageNet Omniglot Aircraft CU Birds DTD Quick Draw Fungi VGG Flower Traffic Sign MSCOCO MNIST
Benchmarks: Meta-Dataset
Metrics: mean accuracy 95% confidence interval average rank running time
Baselines: Finetune ProtoNets BOHB FP-MAML ALFA+FP-MAML FLUTE SSL-HSIC URL ProtoMAML CNAPS S-CNAPS SUR
Builds on: URL SSL-HSIC HSIC Proto-MAML Adadelta
Closest papers by meaning
- Discriminative Sample-Guided and Parameter-Efficient Feature Space Adaptation for Cross-Domain Few-Shot Learning (CVPR 2024 · similarity 0.89)
- Learning To Affiliate: Mutual Centralized Learning for Few-Shot Classification (CVPR 2022 · similarity 0.87)
- Accelerating Convergence in Bayesian Few-Shot Classification (ICML 2024 · similarity 0.85)
- Pushing the Limit of Fine-Tuning for Few-Shot Learning: Where Feature Reusing Meets Cross-Scale Attention (AAAI 2024 · similarity 0.85)
- Mind the Gap Between Prototypes and Images in Cross-domain Finetuning (NeurIPS 2024 · similarity 0.84)
- Flatten Long-Range Loss Landscapes for Cross-Domain Few-Shot Learning (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.