Research map / Label-efficient and robust learning / Recognition and long-tail
Manifold-Based Verbalizer Space Re-embedding for Tuning-Free Prompt-Based Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LLE-INC, a tuning-free manifold-based re-embedding of frozen [MASK] embeddings for prompt-based classification, using locally linear embedding with intra-class neighbor constraints and kNN, optionally combined with contrastive tuning.
Problem
Prompt-based classification with verbalizer embeddings typically requires tuning PTM parameters or extra trainable embeddings, and Euclidean distances between high-dimensional verbalizer embeddings may ignore non-linear manifold structure.
Approach
Re-embed frozen [MASK] output embeddings with locally linear embedding where each instance is reconstructed only from its intra-class nearest neighbors (intra-class neighborhood constraint), then classify test instances with cosine kNN in the re-embedded Euclidean space, requiring no parameter updates.
Key results
- Tuning-free LLE-INC on RoBERTa-large reaches 92.9 on SST-2 and 69.2 on MNLI, comparable to tuned ManualVerb and GenVerb.
- With contrastive tuning, LLE-INC improves up to 3.2% on CHIP-CTC and 1.1% on average across 10 datasets.
- Tuning-free LLaMA-65B with LLE-INC reaches 83.4 on RTE and 84.3 on QNLI, outperforming tuned RoBERTa-large LLE-INC on several tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LLE-INC instead of the strongest listed verbalizer baseline | RoBERTa-Large prompt-based classification · SST-2, MRPC, QQP, MNLI, QNLI, RTE, CHIP-CTC, cMedTC, Kuake-QIC, Tnews | accuracy/F1 | Mixed |
| Add LLE-INC manifold re-embedding (ablation) | LLaMA-7B, LLaMA-13B and LLaMA-65B · SST-2 | accuracy: 88.0 -> 93.2; 88.8 -> 93.8; 86.8 -> 92.8 | Helped |
| Add LLE-INC manifold re-embedding (ablation) | LLaMA-7B, LLaMA-13B and LLaMA-65B · MRPC, QQP, MNLI, QNLI, RTE | accuracy | Helped |
| Replace original LLE neighborhoods with the intra-class neighborhood constraint (ablation) | LLE-INC† · SST-2 | accuracy: 79.2 -> 86.6 | Helped |
| Replace original LLE neighborhoods with the intra-class neighborhood constraint (ablation) | LLE-INC† · MRPC, QQP, MNLI, QNLI, RTE, CHIP-CTC, Kuake-QIC | accuracy | Helped |
| Remove space re-embedding (ablation) | LLE-INC† · SST-2 | accuracy: 86.6 -> 66.5 | Hurt |
| Remove space re-embedding (ablation) | LLE-INC† · MRPC, QQP, MNLI, QNLI, RTE, CHIP-CTC, Kuake-QIC | accuracy | Hurt |
| Add contrastive learning (ablation) | LLE-INC · SST-2 | accuracy: 86.6 -> 92.9 | Helped |
| Add LLE-INC space reconstruction (ablation) | contrastive learning · SST-2 | accuracy: 85.1 -> 92.9 | Helped |
| Add contrastive learning and LLE-INC together rather than either alone (ablation) | RoBERTa-Large prompt-based classification · MRPC, QQP, MNLI, QNLI, RTE, CHIP-CTC, Kuake-QIC, Tnews | accuracy | Helped |
Surprising findings
- Tuning-free LLE-INC on frozen RoBERTa-large is on par with or better than tuned GenVerb and ProtoVerb on several tasks.
- Manifold re-embedding without any tuning substantially improves over plain kNN on original embeddings (e.g., SST-2 66.5 to 86.6).
Limitations
- Re-embedded dimension is a hyperparameter ranging 20 to 400 that must be selected.
- Gains from contrastive tuning are reported with statistical significance only in most cases, and results are from few-shot settings with 5 seeds.
- Evaluated mainly on few-shot settings with a single RoBERTa-large and LLaMA backbones.
Experimental setup
Datasets: SST-2 MRPC QQP MNLI QNLI RTE CHIP-CTC cMedTC KUAKE-QIC Tnews
Benchmarks: SST-2 MRPC QQP MNLI QNLI RTE CHIP-CTC cMedTC Kuake-QIC Tnews
Metrics: accuracy F1
Baselines: ManualVerb GenVerb ProtoVerb Fine-tuning LLE w/o re-embedding
Builds on: Locally Linear Embedding prompt-based tuning LM-BFF ProtoVerb InfoNCE
Closest papers by meaning
- Token Embeddings Violate the Manifold Hypothesis (NeurIPS 2025 · similarity 0.85)
- Leviathan: Decoupling Input and Output Representations in Language Models (NeurIPS 2026 · similarity 0.85)
- ADePT: Adaptive Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning (ICLR 2025 · similarity 0.84)
- EmbedLLM: Learning Compact Representations of Large Language Models (ICLR 2025 · similarity 0.84)
- Hyperbolic Fine-Tuning for Large Language Models (NeurIPS 2025 · similarity 0.84)
- Crafting Interpretable Embeddings for Language Neuroscience by Asking LLMs Questions (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.