Research map / Label-efficient and robust learning / Recognition and long-tail
De-biased Attention Supervision for Text Classification with Causality
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes De-biased Attention Supervision (DAS), using causal backdoor adjustment and direct-effect subtraction to remove label-caused and word-caused biases from keyword-based attention supervision for text classification.
Problem
Attention supervision (AS) for text classification produces more interpretable attention but can hurt classification accuracy in specialized domains, due to label-distribution and word occurrence-range biases.
Approach
Uses a causal graph to identify two biases in keyword-based AS; applies backdoor adjustment over per-label query-specific attention distributions to remove label-caused bias, and reduces word-caused bias by subtracting the word's natural direct effect (mean-pooled attention) from the total effect.
Key results
- On the Legal Verdict dataset, BiLSTM+DAS reaches 86.33% accuracy and 82.22 Ma-F1 versus 85.36 and 79.76 for BiLSTM+AS.
- On Medical Triage, Transformer+DAS improves Ma-F from 63.51 to 65.17 over AS (BiLSTM variant) and low-frequency-label Ma-F rises from 57.48 to 60.16 for Transformer.
- In human evaluation, DAS attention distributions were preferred in 70.63% (legal) and 64.58% (medical) of valid cases.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply DAS instead of keyword-based AS | BiLSTM · Legal Verdict | Acc: 85.36 -> 86.33 | Helped |
| Apply DAS instead of keyword-based AS | BiLSTM · Legal Verdict | Ma-F: 79.76 -> 82.22 | Helped |
| Apply DAS instead of keyword-based AS | BiLSTM · Medical Triage | Acc: 77.33 -> 77.34 | No clear effect |
| Apply DAS instead of keyword-based AS | BiLSTM · Medical Triage | Ma-F: 63.51 -> 65.17 | Helped |
| Add the word-caused bias reduction operation (ablation) | DAS with BiLSTM · Legal Verdict | Acc: 85.81 -> 86.33 | Helped |
| Add the word-caused bias reduction operation (ablation) | DAS with BiLSTM · Legal Verdict | Ma-F: 80.98 -> 82.22 | Helped |
| Add the word-caused bias reduction operation (ablation) | DAS with BiLSTM · Medical Triage | Acc: 77.25 -> 77.34 | Helped |
| Add the word-caused bias reduction operation (ablation) | DAS with BiLSTM · Medical Triage | Ma-F: 64.04 -> 65.17 | Helped |
| Apply backdoor adjustment for label-caused bias instead of standard AS (ablation) | BiLSTM attention supervision · Legal Verdict | Acc: 85.36 -> 85.81 | Helped |
| Apply backdoor adjustment for label-caused bias instead of standard AS (ablation) | BiLSTM attention supervision · Legal Verdict | Ma-F: 79.76 -> 80.98 | Helped |
| Apply backdoor adjustment for label-caused bias instead of standard AS (ablation) | BiLSTM attention supervision · Medical Triage | Acc: 77.33 -> 77.25 | Hurt |
| Apply backdoor adjustment for label-caused bias instead of standard AS (ablation) | BiLSTM attention supervision · Medical Triage | Ma-F: 63.51 -> 64.04 | Helped |
Surprising findings
- Keyword-based attention supervision lowered classification performance compared with unsupervised attention on both professional datasets, despite producing more reasonable attention.
- Different keyword selection methods (TF-IDF, YAKE, AdaKeyBERT, VMask) yielded similar AS effects, so the choice of keyword method was not critical.
Limitations
- Evaluated only on two Chinese professional datasets (legal and medical).
- Human evaluation relies on a subset of cases and annotator judgment of attention rationality.
- Performance dips observed in keyword vocabulary size sweeps suggest sensitivity to keyword quality.
Experimental setup
Datasets: Legal Verdict (CAIL2018) Medical Triage (Chinese medical dialogue data)
Benchmarks: Legal Verdict (CAIL2018) Medical Triage
Metrics: Accuracy Macro-P Macro-R Macro-F1 low25% Ma-F human preference of attention distributions
Baselines: BoW SVM Transformer Transformer+ATT Transformer+AS BiLSTM BiLSTM+ATT BiLSTM+AS
Builds on: Attention Supervision (keyword-based) backdoor adjustment Transformer BiLSTM
Closest papers by meaning
- Rethinking the Development of Large Language Models from the Causal Perspective: A Legal Text Prediction Case Study (AAAI 2024 · similarity 0.84)
- Towards Understanding the Word Sensitivity of Attention Layers: A Study via Random Features (ICML 2024 · similarity 0.84)
- SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking (NeurIPS 2026 · similarity 0.84)
- Classifier-to-Bias: Toward Unsupervised Automatic Bias Detection for Visual Classifiers (CVPR 2025 · similarity 0.84)
- Towards Causal Foundation Model: on Duality between Optimal Balancing and Attention (ICML 2024 · similarity 0.83)
- Causal Prompting: Debiasing Large Language Model Prompting Based on Front-Door Adjustment (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.