Research map / Label-efficient and robust learning / Recognition and long-tail
NOMATTERXAI: Generating “No Matter What” Alterfactual Examples for Explaining Black-Box Text Classification Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NoMatterXAI, a greedy algorithm that generates "alterfactual" text examples by replacing irrelevant words with semantic-field opposites (via ConceptNet or ChatGPT) while keeping the classifier's prediction stable, and validates it with automatic metrics and a MTurk user study.
Problem
Counterfactual explanations focus on relevant features and ignore irrelevant ones, yet irrelevant features (e.g., gender or race words) matter for checking bias; no computational method existed for generating alterfactual examples for text classifiers.
Approach
Formulate alterfactual generation as constrained optimization and solve it greedily: rank words by low importance, replace each with an opposite word from the same semantic field (ConceptNet relations or ChatGPT-3.5 antonyms), and accept replacements only if prediction confidence shifts by at most 5%, context similarity (USE) is at least 0.8, and no double negatives arise.
Key results
- NoMatterXAI finds alterfactual examples for roughly 70% to 95% of inputs across DistilBERT, BERT and RoBERTa.
- Average confidence shift after perturbation is about 0.7% to 0.8% across models.
- In the MTurk study, all three pairwise bias rankings were identified above chance (one-sample t-test significant, e.g., H3 with p=4.5e-10).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NoMatterXAI with ConceptNet replacements instead of Feng et al. | text alterfactual generation · GB, HS, JIG, and EMO | FID: 95.56 -> 77.78; 99.17 -> 92.26; 97.29 -> 89.79; 98.65 -> 95.68 | Hurt |
| Replace words with ChatGPT candidates rather than ConceptNet candidates | NoMatterXAI · GB, HS, JIG, and EMO | SIM: 0.86 -> 0.86; 0.87 -> 0.88; 0.92 -> 0.92; 0.89 -> 0.88 | Mixed |
| Remove words instead of replacing them with context-preserving opposite words | Feng et al. baseline versus NoMatterXAI · Text classification datasets | SIM | Hurt |
| Set window size w to 1 instead of w = 3 (ablation) | Double-negative detection · 150-sentence double-negative dataset | F1: 0.9155 -> 0.7541 | Hurt |
| Set window size w to 2 instead of w = 3 (ablation) | Double-negative detection · 150-sentence double-negative dataset | F1: 0.9155 -> 0.9143 | No clear effect |
| Set window size w to 4 instead of w = 3 (ablation) | Double-negative detection · 150-sentence double-negative dataset | F1: 0.9155 -> 0.9091 | No clear effect |
| Set window size w to 5, 6, or 7 instead of w = 3 (ablation) | Double-negative detection · 150-sentence double-negative dataset | F1: 0.9155 -> 0.9028 | No clear effect |
| Set probability threshold n_t to 0.05, 0.1, or 0.15 instead of n_t = 0.20 (ablation) | Double-negative detection · 150-sentence double-negative dataset | F1: 0.9078 -> 0.9155 | Helped |
| Set probability threshold n_t to 0.25 instead of n_t = 0.20 (ablation) | Double-negative detection · 150-sentence double-negative dataset | F1: 0.9078 -> 0.8921 | Hurt |
| Set probability threshold n_t to 0.30 instead of n_t = 0.20 (ablation) | Double-negative detection · 150-sentence double-negative dataset | F1: 0.9078 -> 0.8675 | Hurt |
| Set probability threshold n_t to 0.35 instead of n_t = 0.20 (ablation) | Double-negative detection · 150-sentence double-negative dataset | F1: 0.9078 -> 0.8333 | Hurt |
| Set probability threshold n_t to 0.40 instead of n_t = 0.20 (ablation) | Double-negative detection · 150-sentence double-negative dataset | F1: 0.9078 -> 0.7581 | Hurt |
Surprising findings
- The Feng et al. baseline reaches higher fidelity (about 95-99%) than NoMatterXAI but discards much of the original context, giving much lower similarity and higher perplexity.
Limitations
- ConceptNet coverage is limited and many queries return no result.
- Generation is slower than counterfactual methods because it perturbs many words.
- ChatGPT occasionally hallucinates or returns output in the wrong format.
Experimental setup
Datasets: gender bias (GB) hate speech (HS) emotion (EMO) Jigsaw unintended bias (JIG)
Benchmarks: GB HS EMO JIG
Metrics: fidelity average words perturbed average queries altered perplexity semantic similarity (USE) confidence shift
Baselines: Feng et al. word-removal explanation method
Builds on: Mertes et al. alterfactual explanations counterfactual explanations for text ConceptNet
Closest papers by meaning
- A General Search-Based Framework for Generating Textual Counterfactual Explanations (AAAI 2024 · similarity 0.88)
- DiCoFlex: Model-Agnostic Diverse Counterfactuals with Flexible Control (NeurIPS 2025 · similarity 0.86)
- Towards Unifying Evaluation of Counterfactual Explanations: Leveraging Large Language Models for Human-Centric Assessments (AAAI 2025 · similarity 0.85)
- Counterfactual Explanations for Misclassified Images: How Human and Machine Explanations Differ (Abstract Reprint) (AAAI 2024 · similarity 0.85)
- Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals (ICLR 2024 · similarity 0.85)
- Robust Stochastic Graph Generator for Counterfactual Explanations (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.