Research map / Label-efficient and robust learning / OOD and anomaly detection
Meta-Crafting: Improved Detection of Out-of-Distributed Texts via Crafting Metadata Space (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Meta-Crafting, which combines seven model-driven metadata signals into a feature space scored with Mahalanobis distance to detect both background and semantic text OOD shifts.
Problem
Single-score OOD detectors (MSP, perplexity) work inconsistently across background (style) and semantic (content) shifts in NLP, and fail on stress/challenge datasets built from small perturbations.
Approach
Concatenate seven model-driven metadata (average confidence, confidence variability, BALD, MC-BALD, kNN sentence density, MSP, PPL) into one multi-dimensional feature space and compute class-conditional Mahalanobis distance as the OODness score.
Key results
- Meta-Crafting reaches 73.7 AUROC on IMDB vs c-IMDB (semantic shift), versus 53.5 for PPL and 63.7 for MSP.
- On MNLI vs HANS, Meta-Crafting scores 92.4 AUROC against 98.3 for PPL, so it does not win every setting.
- The authors report an average improvement of 101% over PPL and 39% over MSP across challenge datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine seven model-driven metadata using Mahalanobis distance instead of a single-score method | OOD detection with RoBERTa and GPT-2 metadata · Challenge datasets in Table 1 | AUROC: Meta: 73.7; PPL: 53.5; MSP: 63.7 (c-IMDB); Meta: 84.4; PPL: 44.5; MSP: 60.5 (Negation); Meta: 82.3; PPL: 19.6; MSP: 51.6 | Helped |
| Combine seven model-driven metadata using Mahalanobis distance instead of a single-score method | OOD detection with RoBERTa and GPT-2 metadata · HANS | AUROC: PPL: 98.3; Meta: 92.4 | Hurt |
Surprising findings
- PPL outperforms Meta-Crafting on the HANS subset of MNLI (98.3 vs 92.4 AUROC), showing the combined method is not uniformly best.
Limitations
- Some metadata (e.g., the current set) fails to detect certain challenge data, as the authors acknowledge.
- Results are reported as a student abstract with limited experiments and no variance reported beyond three seeds for fine-tuning.
- Metadata selection relies on empirical choice via regression coefficients from prior work.
Experimental setup
Datasets: IMDB MNLI c-IMDB HANS
Benchmarks: IMDB vs c-IMDB MNLI vs HANS and stress test shifts
Metrics: AUROC
Baselines: MSP PPL
Builds on: Mahalanobis distance OOD detection (Lee et al. 2018) Arora, Huang, and He 2021 text OOD benchmark Ren et al. 2019 background vs semantic decomposition Kim et al. 2023 metadata selection
Closest papers by meaning
- Human Texts Are Outliers: Detecting LLM-generated Texts via Out-of-distribution Detection (NeurIPS 2025 · similarity 0.87)
- The Best of Both Worlds: On the Dilemma of Out-of-distribution Detection (NeurIPS 2024 · similarity 0.84)
- ImageNet-OOD: Deciphering Modern Out-of-Distribution Detection Algorithms (ICLR 2024 · similarity 0.84)
- Unsupervised Layer-Wise Score Aggregation for Textual OOD Detection (AAAI 2024 · similarity 0.84)
- DALD: Improving Logits-based Detector without Logits from Black-box LLMs (NeurIPS 2024 · similarity 0.84)
- Training-free LLM-generated Text Detection by Mining Token Probability Sequences (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.