Research map / Label-efficient and robust learning / Recognition and long-tail
RMIB: Representation Matching Information Bottleneck for Matching Text Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RMIB, an information-bottleneck extension for asymmetric-domain text matching that aligns text representations to a Gaussian prior and maximizes interaction between the two representations, improving four matching models on five datasets.
Problem
Asymmetric-domain text matching: how to align text representation distributions across domains and what role this alignment plays; existing IB does not constrain interaction between the two text representations.
Approach
Narrow each text representation's distribution by matching it to a shared standard Gaussian prior, showing this equals optimizing the information bottleneck; extend IB with interaction and inadequacy constraints so the objective becomes maximizing conditional mutual information I(Z1;Z2|Y), which is approximated via a MINE-style bound.
Key results
- SBERT with RMIB improves accuracy on SICK from 72.05 to 84.59 (+12.54 points).
- BERT with RMIB improves SICK accuracy from 86.69 to 87.40 and SciTail accuracy from 93.79 to 94.12.
- RMIB consistently outperforms the IB ablation (alpha2 and alpha3 set to 0) across datasets and models, per the paper's ablation table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add RMIB to BERT | BERT · SICK | accuracy: 86.69 -> 87.40 | Helped |
| Add RMIB to BERT | BERT · SciTail | accuracy: 93.79 -> 94.12 | Helped |
| Add RMIB to BERT | BERT · WikiQA | F1: 94.88 -> 95.16 | Helped |
| Add RMIB to BERT | BERT · WikiQA | MAP: 83.40 -> 84.29 | Helped |
| Add RMIB to BERT | BERT · WikiQA | MRR: 85.01 -> 86.02 | Helped |
| Add RMIB to BERT | BERT · SNLI | accuracy: 90.46 -> 90.43 | Hurt |
| Add RMIB to SBERT | SBERT · Quora Question Pair | accuracy: 86.46 -> 87.34 | Helped |
| Add RMIB to BERT | BERT · MRPC | accuracy: 85.54 -> 87.01 | Helped |
| Add RMIB to BERT | BERT · WNLI | accuracy: 53.52 -> 56.34 | Helped |
| Add RMIB to BERT | BERT · RTE | accuracy: 62.82 -> 69.31 | Helped |
| Remove the prior-distribution KL term (set α1 to 0) (ablation) | RMIB · multiple text matching datasets | accuracy | Mixed |
| Remove the representation-interaction term (set α2 to 0) (ablation) | RMIB · multiple text matching datasets | accuracy | Mixed |
Surprising findings
- Ablating the alpha1 term in BERT slightly improves Quora accuracy (+0.16), and removing components sometimes helps on individual datasets.
- BERT accuracy on SNLI drops by 0.03 with RMIB, while SBERT gains on every dataset including WikiQA where it gains only 0.88 MRR-level improvement.
Limitations
- Several hyper-parameters (alpha1, alpha2, alpha3) must be set manually; search range limited to {0.01, 0.02, 0.03}.
- Gains are smaller on large datasets such as SNLI and Quora, and BERT on SNLI does not improve.
Experimental setup
Datasets: SICK SciTail WikiQA SNLI Quora Question Pairs MRPC RTE WNLI
Benchmarks: SICK SciTail WikiQA SNLI Quora Question Pairs MRPC RTE WNLI
Metrics: accuracy F1 MAP MRR
Baselines: RE2 BERT SBERT ESIM DDR-Match (JS) DDR-Match (MMD) DDR-Match (WD) IB (RMIB ablation)
Builds on: information bottleneck (Tishby et al., 2000) VIB DDR-Match MINE RE2 ESIM
Closest papers by meaning
- Expanding the Scope of Negatives: Boosting Image-Text Matching with Negatives Distribution Guided Learning (AAAI 2025 · similarity 0.84)
- Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe (NeurIPS 2024 · similarity 0.84)
- SoftMatcha: A Soft and Fast Pattern Matcher for Billion-Scale Corpus Searches (ICLR 2025 · similarity 0.83)
- Achieving Ensemble-Like Performance in a Single Model: A Feature Diversification Framework for Image-Text Matching (AAAI 2025 · similarity 0.83)
- Narrowing Information Bottleneck Theory for Multimodal Image-Text Representations Interpretability (ICLR 2025 · similarity 0.83)
- MASS: Overcoming Language Bias in Image-Text Matching (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.