Research map / Architectures and efficiency / RNNs and normalization
Breaking Free from MMI: A New Frontier in Rationalization by Probing Input Utilization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes N2R, which replaces the MMI (cross-entropy) rationale-extractor objective with maximizing the norm of the predictor encoder's representation of the rationale candidate, and shows gains over RNP and MMI variants on beer/hotel text, BA2Motifs, CUB, and MovieReview.
Problem
Rationale extraction with the MMI criterion suffers from diminishing marginal returns: once part of the true rationale is selected, adding the rest barely reduces cross-entropy, so gradients fail to locate the remaining rationale tokens.
Approach
Instead of reproducing the prediction, the extractor is trained to maximize the L2 norm of the predictor encoder's representation of the rationale candidate, based on the idea that inputs the network can utilize (lying in the weight matrix's capability subspace) yield high-norm representations, while unlearned inputs yield near-zero norms; this norm objective can also be combined with MMI.
Key results
- On Beer-Appearance, N2R reaches 72.7 F1 versus 66.5 for RNP and 67.1 for the best prior variant (A2I); on Hotel-Cleanliness it reaches 39.8 F1 versus 35.6 for RNP.
- With BERT encoder on Beer-Appearance, N2R attains 49.9 F1 versus 39.0 for CR.
- Combining N2R with MMI (MMI+N2R) raises F1 relative to vanilla MMI by 13.1% on Beer-Appearance, and N2R converges faster than RNP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the extractor's cross-entropy objective with representation norm | RNP rationalization with GRU encoders · Beer-Appearance, Beer-Aroma, Hotel-Service, and Hotel-Cleanliness | F1: 66.5 -> 72.7; 65.8 -> 68.7; 46.8 -> 48.6; 35.6 -> 39.8 | Helped |
| Replace the extractor's cross-entropy objective with representation norm | RNP rationalization with bert-base-uncased · Beer-Appearance and Beer-Aroma | F1: 39.0 -> 49.9; 49.1 -> 57.6 | Helped |
| Replace the extractor's cross-entropy objective with representation norm | RNP with GCN · BA2Motifs | F1: 36.4 -> 40.2 | Helped |
| Replace the extractor's cross-entropy objective with representation norm | RNP with U-Net extractor and ResNet18 predictor · CUB | F1: 35.0 -> 49.1 | Helped |
| Replace the extractor's cross-entropy objective with representation norm | RNP with GRU and GloVe-100d · MovieReview | F1: 24.1 -> 37.4 | Helped |
| Add the representation norm objective to the extractor's MMI objective (ablation) | RNP rationalization · Beer-Appearance, Beer-Aroma, Hotel-Service, and Hotel-Cleanliness | F1: +13.1%; +8.4%; +5.8%; +13.5% | Helped |
| Use representation norm rather than prediction accuracy or cross-entropy to score rationale candidates | A trained standard RNP predictor · Beer-Aroma rationale candidates with increasing proportions of gold-rationale tokens | representation norm | Helped |
| Increase the proportion of true rationale components beyond 60% | A trained standard RNP predictor · Beer-Aroma | accuracy | No clear effect |
Surprising findings
- The norm-based extractor objective alone outperforms the MMI objective, and N2R's accuracy on Beer-Aroma and Beer-Appearance is at or above a BERT classifier, suggesting extracted rationales can improve classification.
- The llama-3.1-8b fine-tuned model is competitive on some datasets, but N2R beats it on two of four text datasets and is comparable on the remaining two.
Limitations
- Theoretical argument relies on a simplifying uniform-informativeness assumption that may not hold in practice.
- Hyperparameters tuned for RNP and reused for N2R; some baselines (CR) copied from original papers and hyperparameter-sensitive methods were not extensively tuned with BERT.
- Graph and image experiments are supplementary and limited in scope; interpretability for GNNs left for future work.
Experimental setup
Datasets: BeerAdvocate (Beer-Appearance, Beer-Aroma) HotelReviews (Hotel-Service, Hotel-Cleanliness) BA2Motifs CUB MovieReview
Benchmarks: BeerAdvocate HotelReviews BA2Motifs CUB MovieReview
Metrics: F1 precision recall sparsity S accuracy IoU
Baselines: RNP Inter_RAT NIR A2I CR A2R INVRAT llama-3.1-8b-instruct
Builds on: RNP (rationalizing neural predictions) Inter_RAT A2I CR NIR
Closest papers by meaning
- Is the MMI Criterion Necessary for Interpretability? Degenerating Non-causal Features to Plain Noise for Self-Rationalization (NeurIPS 2024 · similarity 0.89)
- Towards Faithful Explanations: Boosting Rationalization with Shortcuts Discovery (ICLR 2024 · similarity 0.86)
- Learning Robust Rationales for Model Explainability: A Guidance-Based Approach (AAAI 2024 · similarity 0.84)
- Adversarial Cooperative Rationalization: The Risk of Spurious Correlations in Even Clean Datasets (ICML 2025 · similarity 0.84)
- Rethinking Dimensional Rationale in Graph Contrastive Learning from Causal Perspective (AAAI 2024 · similarity 0.84)
- Local vs. Global Interpretability: A Computational Complexity Perspective (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.