Research map / Label-efficient and robust learning / Recognition and long-tail
Know2Vec: A Black-Box Proxy for Neural Network Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Know2Vec, a black-box proxy that encodes decision-boundary knowledge of neural network models into vectors via LSTMs and aligns them with query-task vectors for neural network retrieval.
Problem
Choosing a suitable pre-trained model from a model zoo for a user's query dataset without accessing model parameters or training data, with accurate and privacy-preserving model vectorization and query-model alignment.
Approach
Abstracts each model's decision knowledge from centroid and decision-boundary samples (obtained via black-box queries with probe data) into a graph encoded by inner/outer bidirectional LSTMs, maps query samples to a task vector with an LSTM, and aligns both in a cosine space trained with a model-classification cross-entropy loss plus a margin-based cosine alignment loss.
Key results
- On the Kaggle NNR task, Know2Vec reaches R@1 94.82% and fine-tuned accuracy 95.67%, versus 82.75% R@1 for TANS (the strongest baseline).
- On SF-MTE classification tasks, Know2Vec gets the best Pearson and Spearman on DTD (0.4942/0.5122) and Aircraft (0.5545/0.5779), but is not best on all regression metrics.
- Ablation on query samples shows accuracy rising with more query images per class (e.g., +11.03% from 2 to 8 images for 128-d vectors).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Know2Vec instead of Leep for model retrieval | Kaggle model hub NNR · Kaggle model hub NNR | R@1: 93.10 -> 94.82 | Helped |
| Use Know2Vec instead of TANS for model retrieval | Kaggle model hub NNR · Kaggle model hub NNR | Ft. Acc: 94.22 -> 95.67 | Helped |
| Use Know2Vec instead of LFC | SF-MTE · DTD | Pearson correlation: 0.3508 -> 0.4942 | Helped |
| Use Know2Vec instead of LogME | SF-MTE · Aircraft | Spearman correlation: 0.5337 -> 0.5779 | Helped |
| Use Know2Vec instead of LFC | SF-MTE · Mean | Pearson correlation: 0.3915 -> 0.3576 | Hurt |
| Use Know2Vec instead of the strongest baseline | SF-MTE · dSprites | Pearson correlation: 0.2243 -> 0.1917 | Hurt |
| Replace averaging with an LSTM query encoder (ablation) | Know2Vec on Kaggle model hub NNR · Kaggle model hub NNR | retrieval accuracy: 92.54% -> 94.82% | Helped |
| Replace concatenation with an LSTM query encoder (ablation) | Know2Vec on Kaggle model hub NNR · Kaggle model hub NNR | retrieval accuracy: 88.14% -> 94.82% | Helped |
| Use cosine spatial alignment loss instead of the tested alternatives (ablation) | Know2Vec on Kaggle model hub NNR · Kaggle model hub NNR | retrieval accuracy: 94.82% | Helped |
| Add model embedding consistency loss L_MKC (ablation) | Know2Vec on Kaggle model hub NNR · Kaggle model hub NNR | retrieval accuracy | Helped |
| Increase query samples per class from 2 to 8 (ablation) | Know2Vec on Kaggle model hub NNR · Kaggle model hub NNR | retrieval accuracy: 11.03%, 10.86%, and 8.79% | Helped |
| Vary query vector length among 128, 256, and 512 dimensions (ablation) | Know2Vec on Kaggle model hub NNR · Kaggle model hub NNR | retrieval accuracy: 5.18% | Mixed |
Surprising findings
- Model Spider, LogME and dynamic SF-MTE methods underperform on top-1 NNR, which the authors attribute to focus on ranking rather than top-1 selection.
- GPT-4 and Gemini achieve negative or near-zero correlations on most SF-MTE tasks.
Limitations
- SF-MTE ground truth for training uses partial (head-only) fine-tuning as an approximation of full fine-tuning.
- Performance is not best across all SF-MTE regression metrics; authors attribute this to training only on classification tasks.
- Probe dataset coverage limits boundary sample quality when only a subset of categories is available.
Experimental setup
Datasets: Kaggle model zoo (58 classification models, 232 testing tasks) Pre-trained model zoo (48 models from Inception_V3, ResNet50, DenseNet201 on 16 datasets) CUB2011 CIFAR100 SUN397 Dogs VLCS ImageNet OfficeHome Aircraft DTD UTKFace
Benchmarks: Kaggle model zoo NNR tasks PTM zoo SF-MTE on Aircraft, DTD, UTKFace, dSprites
Metrics: R@1 R@3 valid accuracy fine-tuned accuracy search time Pearson correlation Spearman correlation
Baselines: H-Score NCE LEEP NLeep LFC LogME Model Spider TANS GPT-4 Gemini
Builds on: TANS Model Spider DNNR LogME
Closest papers by meaning
- Think-on-Graph 2.0: Deep and Faithful Large Language Model Reasoning with Knowledge-guided Retrieval Augmented Generation (ICLR 2025 · similarity 0.83)
- Simple is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generation (ICLR 2025 · similarity 0.83)
- Neural Architecture Retrieval (ICLR 2024 · similarity 0.82)
- Exploiting Distribution Constraints for Scalable and Efficient Image Retrieval (ICLR 2025 · similarity 0.82)
- MUVERA: Multi-Vector Retrieval via Fixed Dimensional Encoding (NeurIPS 2024 · similarity 0.82)
- GFM-RAG: Graph Foundation Model for Retrieval Augmented Generation (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.