Research map / Vision-language and multimodal / Human and embodied modeling

Contradiction Retrieval via Contrastive Learning with Sparsity

ICML 2025 · Vision-language and multimodal › Human and embodied modeling · approach: Multimodal retrieval-augmented modeling

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SparseCL, a contrastive fine-tuning of sentence embeddings that rewards Hoyer sparsity of embedding differences, combined with cosine similarity to retrieve contradicting passages.

Problem

Retrieving documents that explicitly contradict a query from a large corpus; cosine similarity search cannot represent contradiction relations and cross-encoders are too slow.

Approach

Fine-tune sentence embeddings so that contradicting passages differ in only a few coordinates (high Hoyer sparsity of the difference vector), and score candidates by cosine similarity plus a weighted Hoyer sparsity term, avoiding costly cross-encoder inference.

Key results

• On contradiction retrieval, zeroshot BGE with cosine plus SparseCL Hoyer reaches 96.7 NDCG@10 on HotpotQA and 90.9 on MSMARCO versus 59.5 and 60.0 for cosine alone.
• GTE with CL (Cosine) + SparseCL (Hoyer) reaches 81.3 NDCG@10 on Arguana versus 72.5 for zeroshot GTE cosine.
• On corpus cleaning, filtering contradictions recovers over 60% of the QA performance loss and reduces corruption ratio below 5%.

Measured effects reported in the paper

Surprising findings

• Using Hoyer sparsity alone to retrieve contradictions performs poorly (56.1 on Arguana), suggesting contradiction requires both similarity and dissimilarity.
• Standard contrastive learning with cosine similarity makes contradiction and entailment pairs almost indistinguishable on SNLI/MNLI.

Limitations

• Arguana is the only public contradiction-style dataset; MSMARCO and HotpotQA contradictions are synthetic GPT-4 generations.
• Requires a tuned weight alpha per dataset and notion of contradiction.
• Sparsity can be degraded when many paraphrases are in the corpus unless combined with cosine similarity; fixed Hoyer alone underperforms.

Experimental setup

Datasets: Arguana MSMARCO HotpotQA SNLI MNLI

Benchmarks: Arguana MSMARCO (modified) HotpotQA (modified) SNLI MNLI

Metrics: NDCG@10 Recall@10 average cosine / Hoyer score

Baselines: Zeroshot (Cosine) CL (Cosine) Prompt + CL (Cosine) Gen + CL (Cosine) cross-encoder Bipolar-encoder (Shi et al. 2023)

Builds on: contrastive learning for sentence embeddings (SimCSE) Hoyer sparsity measure (Hurley & Rickard 2009) counter-argument retrieval (Wachsmuth et al. 2018)

Closest papers by meaning

• Mitigating the Impact of False Negative in Dense Retrieval with Contrastive Confidence Regularization (AAAI 2024 · similarity 0.84)
• PhantomWiki: On-Demand Datasets for Reasoning and Retrieval Evaluation (ICML 2025 · similarity 0.84)
• Perplexity Trap: PLM-Based Retrievers Overrate Low Perplexity Documents (ICLR 2025 · similarity 0.84)
• Worse than Zero-shot? A Fact-Checking Dataset for Evaluating the Robustness of RAG Against Misleading Retrievals (NeurIPS 2025 · similarity 0.83)
• OBLIQ-Bench: Exposing Overlooked Bottlenecks in Modern Retrievers with Latent and Implicit Queries (NeurIPS 2026 · similarity 0.83)
• Making Retrieval-Augmented Language Models Robust to Irrelevant Context (ICLR 2024 · similarity 0.83)