Research map / Large language models / Language, safety and interpretability
Labels Need Prompts Too: Mask Matching for Natural Language Understanding Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Mask Matching, which applies prompting to both input and label sides (input-prompt and label-prompt mask representations matched via cross-entropy) and evaluates it on 14 NLU datasets across 8 tasks.
Problem
Classification-based fine-tuning, semantic matching, and prompt-tuning under-utilize the semantic information in textual label names for NLU tasks, especially those with many labels.
Approach
Add a label-prompt with its own mask token to each label name, and classify by matching the input-prompt mask representation against all label-prompt mask representations with a softmax cross-entropy, avoiding manual verbalizers while keeping label semantics.
Key results
- Mask Matching improves over fine-tuning on all 14 datasets in full training, e.g., TAC-REV 83.2 to 84.4 and BBN 80.3 to 81.2 (reported as 80.0 to 81.2 in text).
- In the low-resource setting (10% training data), Mask Matching beats prompt-tuning on 12 of 14 datasets, e.g., FEW-NERD 74.7 to 82.2.
- Mask Matching is insensitive to the choice of label-prompt template across four tested templates.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use label-side mask prompts and match label-mask representations to the input-mask representation instead of fine-tuning | NLU tasks under full training · R8, R52, MR, IMDb, FEW-NERD, BBN, TACRED, TAC-REV, QNLI, SNLI, PIT2015, QQP, WiC, VAST | task metric: 0.2% to almost 2% | Helped |
| use Mask Matching instead of Prompt-tuning with 10% of training data | NLU tasks in the low-resource setting · 12 of 14 datasets | task metric | Helped |
| use Mask Matching instead of Prompt-tuning with 10% of training data | QQP · QQP | task metric: +0.0 | No clear effect |
| use Mask Matching instead of Prompt-tuning with 10% of training data | VAST · VAST | Macro-F1: -1.7 | Hurt |
| add two related words to each label name (ablation) | Mask Matching · FEW-NERD | Loose Micro-F1: 85.2 -> 85.6 | Helped |
| add two related words to each label name (ablation) | Mask Matching · TACRED | Micro-F1: 75.2 -> 75.4 | Helped |
| vary the label-prompt template among P1, P2, P3, and P4 (ablation) | Mask Matching · R52, TACRED, QNLI, VAST | task metric | No clear effect |
| replace prompt-based label representations with max-pooling over label names (ablation) | Mask Matching · 14 NLU datasets | task metric | Hurt |
Surprising findings
- Semantic Matching with max-pooling over label names performs poorly on relation classification and entity typing, attributed to word-frequency effects on representations.
- Mask Matching underperforms prompt-tuning on VAST in the low-resource setting (73.8 vs 75.5).
Limitations
- Gains on sentence-pair tasks (NLI, paraphrase) are small because concatenating two sentences with one mask is suboptimal.
- Relies on informative, meaningful label names; label-name augmentation is manual.
- Only one mask token per prompt was explored.
Experimental setup
Datasets: R8 R52 MR IMDb FEW-NERD BBN TACRED TACRED-Revisited QNLI SNLI PIT2015 QQP
Benchmarks: R8 R52 MR IMDb FEW-NERD BBN TACRED TACRED-Revisited QNLI SNLI PIT2015 QQP
Metrics: accuracy Micro-F1 Loose Micro-F1 Macro-F1
Baselines: Fine-tuning Prompt-tuning Semantic Matching
Builds on: prompt-tuning semantic matching RoBERTa BERT
Closest papers by meaning
- Love Me, Love My Label: Rethinking the Role of Labels in Prompt Retrieval for Visual In-Context Learning (CVPR 2026 · similarity 0.86)
- Multi-Mask Label Mapping for Prompt-Based Learning (AAAI 2023 · similarity 0.86)
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (ICLR 2019 · similarity 0.85)
- LAMM: Label Alignment for Multi-Modal Prompt Learning (AAAI 2024 · similarity 0.85)
- Few-Shot Learning Evaluation in Natural Language Understanding (NeurIPS 2021 · similarity 0.85)
- Prompt Sketching for Large Language Models (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.