Research map / Large language models / Language, safety and interpretability
IRM—when it works and when it doesn't: A test case of natural language inference
Yana Dranker, He He, Yonatan Belinkov
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use IRM instead of ERM | IRM vs ERM · toy experiment test | accuracy: 0.0 -> 100.0 | Improved |
| use IRM instead of ERM, synthetic bias pe = 0.0 | IRM vs ERM · SNLI synthetic bias | accuracy: 79.16 -> 83.5 | Improved |
| use IRM instead of ERM, synthetic bias pe = 0.33 | IRM vs ERM · SNLI synthetic bias | accuracy: 85.16 -> 87.22 | Improved |
| use IRM instead of ERM, synthetic bias pe = 0.8 (in-distribution) | IRM vs ERM · SNLI synthetic bias | accuracy: 93.49 -> 92.32 | Hurt |
| use IRM instead of ERM on hypothesis bias, bias misaligned subset | IRM vs ERM · SNLI | accuracy: 62.63 -> 65.12 | Improved |
| use IRM instead of ERM on hypothesis bias, bias aligned subset | IRM vs ERM · SNLI | accuracy: 97.4 -> 91.4 | Hurt |
| use IRM instead of ERM on hypothesis bias, unbiased subset | IRM vs ERM · SNLI | accuracy: 84.46 -> 82.64 | Hurt |
| use IRM instead of ERM on overlap bias, bias misaligned subset | IRM vs ERM · MNLI dev mismatched | F1 macro: 62.66 -> 64.12 | Improved |
| use IRM instead of ERM on overlap bias, bias aligned subset | IRM vs ERM · MNLI dev mismatched | F1 macro: 96.67 -> 95.44 | Hurt |
| evaluate on HANS and RevisedPremises | IRM vs ERM · natural bias models | HANS / RP accuracy | No clear effect |
| increase bias prevalence in training set (ablation) | IRM vs ERM · synthetic and natural bias, bias misaligned subset | accuracy | Improved |
| increase bias strength (ablation) | IRM vs ERM · synthetic bias, bias misaligned subset | accuracy | Mixed |
Closest papers by meaning
- LLM Processes: Numerical Predictive Distributions Conditioned on Natural Language (NeurIPS 2024 · similarity 0.87)
- A Causal Marriage between VLM and IRM from Understanding to Reasoning (CVPR 2026 · similarity 0.86)
- Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuning (ICML 2025 · similarity 0.86)
- Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoning (NeurIPS 2025 · similarity 0.86)
- A Bias-Variance-Covariance Decomposition of Kernel Scores for Generative Models (ICML 2024 · similarity 0.86)
- Connecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Data (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
