Research map / Large language models / Instruction and fine-tuning
Faithfulness Measurable Masked Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes masked fine-tuning of RoBERTa so that token masking is in-distribution, making a model "inherently faithfulness measurable" and enabling cheap erasure-based faithfulness evaluation of importance measures validated with MaSF OOD tests.
Problem
Measuring faithfulness of token importance measures via the erasure (masking) metric is confounded by out-of-distribution inputs created by masking, and existing fixes require expensive retraining or proxy models.
Approach
Fine-tune a masked language model with masked fine-tuning (half of each mini-batch uniformly masked at 0-100% ratio, half unmasked) so that arbitrary masking is in-distribution by design; validate with MaSF non-parametric OOD p-values and measure faithfulness with a Recursive-ROAR-style erasure metric without retraining. Also introduces signed variants of importance measures.
Key results
- Masked fine-tuning does not decrease unmasked performance across 16 datasets, while only-masking causes unstable convergence and plain fine-tuning fails on 100% masked inputs (below class-majority baseline).
- Occlusion-based importance measures (LOO, Beam) are the most faithful; on SST2 Beam signed reaches RACU about 187% versus Recursive ROAR's gradient RACU of 26.1%.
- MaSF in-distribution p-values show masked inputs are consistently in-distribution only with masked fine-tuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add masked fine-tuning to support variable-rate masking (ablation) | RoBERTa fine-tuning · 16 classification datasets | unmasked benchmark performance | No clear effect |
| Use only uniformly masked training data, without an unmasked half (ablation) | RoBERTa fine-tuning · classification benchmarks | unmasked benchmark performance | Hurt |
| Add masked training examples (ablation) | Plain-fine-tuned RoBERTa · 100% masked inputs | performance relative to class-majority baseline | Helped |
| Add masked fine-tuning while retaining an unmasked half of the mini-batch (ablation) | RoBERTa · unmasked and 100% masked inputs | joint unmasked performance and masking support | Helped |
| Use occlusion-based explanations (LOO or Beam) rather than gradient-based explanations | RoBERTa-base with masked fine-tuning · SST2 and bAbI-2 | RACU faithfulness: SST2: Grad (L2) 40.4 vs LOO (abs) 54.9 and Beam 187.3; bAbI-2: Grad (L2) 96.3 vs LOO (abs) 96.3 and Beam 139.2 | Helped |
| Use x ⊙ grad (sign) instead of x ⊙ grad (abs) | RoBERTa-base with masked fine-tuning · SST2 | RACU faithfulness: 23.5 -> -12.2 | Hurt |
| Use signed rather than absolute x ⊙ grad | RoBERTa-base with masked fine-tuning · bAbI-2 | RACU faithfulness: 92.0 -> 65.7 | Hurt |
| Use masked fine-tuning rather than Recursive-ROAR for Grad (L2) explanations | RoBERTa-base · SST2 | RACU faithfulness: 26.1 -> 40.4 | Helped |
| Use masked fine-tuning rather than Recursive-ROAR for Grad (L2) explanations | RoBERTa-base · bAbI-2 | RACU faithfulness: 57.8 -> 96.3 | Helped |
| Apply masking-based faithfulness evaluation without masked fine-tuning (ablation) | RoBERTa with plain fine-tuning · importance-measure faithfulness | faithfulness validity | Hurt |
Surprising findings
- Input times gradient is the worst explanation, while gradient and integrated gradient perform much better.
- Leave-one-out occasionally outperforms the Beam search explanation.
Limitations
- Requires building a faithfulness-measurable model ahead of time; not a post-hoc method.
- Only tested with masked language models (RoBERTa); not generalized to causal LMs.
- In-distribution cannot be statistically proven, and MaSF can flag 100% masked data as OOD for some datasets (e.g., bAbI-2).
Experimental setup
Datasets: RTE SNLI MNLI QNLI CB MRPC QQP SST2 IMDB Anemia Diabetes CoLA
Benchmarks: BoolQ MRPC SST2 bAbI-2 RTE SNLI MNLI QNLI CB QQP IMDB Anemia
Metrics: accuracy macro F1 micro F1 Matthews correlation ACU RACU MaSF in-distribution p-values
Baselines: Recursive ROAR plain fine-tuning only masking
Builds on: ROAR Recursive ROAR MaSF RoBERTa integrated gradients
Closest papers by meaning
- Measuring what Matters: Construct Validity in Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.85)
- Representation Deficiency in Masked Language Modeling (ICLR 2024 · similarity 0.85)
- Walk the Talk? Measuring the Faithfulness of Large Language Model Explanations (ICLR 2025 · similarity 0.85)
- Statistical Hypothesis Testing for Auditing Robustness in Language Models (ICML 2025 · similarity 0.85)
- Scaling up Masked Diffusion Models on Text (ICLR 2025 · similarity 0.84)
- Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.