Research map / Vision-language and multimodal / Human and embodied modeling
RETSim: Resilient and Efficient Text Similarity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RETSim, a lightweight 536k-parameter multilingual metric-learning text embedding trained with typo/adversarial augmentation on mC4 for near-duplicate retrieval and dedup, plus the W4NT3D adversarial multilingual benchmark.
Problem
Near-duplicate text detection is dominated by MinHash, which is parameter-sensitive and brittle to typos and hash-busting; neural embeddings are large and sub-word tokenized, so they are not robust or efficient for dataset deduplication.
Approach
Combine the RETVec character-level vectorizer, a small GAU-based transformer, chunk-level averaging, and Multi-Similarity metric learning with heavy sentence/word/character augmentation to produce a tiny multilingual embedding robust to typos for near-duplicate matching.
Key results
- RETSim Near-Dup achieves average Recall@1 of 0.977 across 41 languages on W4NT3D, versus 0.932 for the best baseline (Multilingual E5-Base).
- RETSim Partial-Dup reaches ARI 0.831 on NEWS-COPY, beating the best MinHash result of 0.783.
- On spam campaign clustering, RETSim Near-Dup achieves V-Measure 0.949 versus 0.903 for USE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RETSim Near-Dup instead of Multilingual E5-Base | W4NT3D · W4NT3D (41-language average) | Recall@1: 0.937 -> 0.977 | Helped |
| Use RETSim Partial-Dup instead of MinHash (Ours) | NEWS-COPY · NEWS-COPY | ARI: 0.783 -> 0.831 | Helped |
| Use RETSim Partial-Dup instead of LaBSE | CORE Near-Duplicates · CORE Near-Duplicates | Macro F1: 0.933 -> 0.945 | Helped |
| Use RETSim Near-Dup instead of USE | Adversarial spam clustering · Adversarial spam campaigns | V-Measure: 0.903 -> 0.949 | Helped |
| Replace GAU blocks with RETVec MLP (ablation) | RETSim Near-Dup · W4NT3D ablation subset | Recall@1: 0.986 -> 0.975 | Hurt |
| Replace GAU blocks with ConvNeXt (ablation) | RETSim Near-Dup · W4NT3D ablation subset | Recall@1: 0.986 -> 0.978 | Hurt |
| Replace GAU blocks with BERT (ablation) | RETSim Near-Dup · W4NT3D ablation subset | Recall@1: 0.986 -> 0.973 | Hurt |
| Replace GAU blocks with T5 (ablation) | RETSim Near-Dup · W4NT3D ablation subset | Recall@1: 0.986 -> 0.980 | Hurt |
| Set chunk size to 128 instead of 512 (ablation) | RETSim Near-Dup · W4NT3D ablation subset | Recall@1: 0.986 -> 0.979 | Hurt |
| Set chunk size to 256 instead of 512 (ablation) | RETSim Near-Dup · W4NT3D ablation subset | Recall@1: 0.986 -> 0.984 | Hurt |
| Set chunk size to 1024 instead of 512 (ablation) | RETSim Near-Dup · W4NT3D ablation subset | Recall@1: 0.986 -> 0.983 | Hurt |
| Set chunk size to 2048 instead of 512 (ablation) | RETSim Near-Dup · W4NT3D ablation subset | Recall@1: 0.986 -> 0.978 | Hurt |
Surprising findings
- RETSim Near-Dup underperforms on NEWS-COPY (ARI 0.704) relative to its Partial-Dup variant, because many positive pairs differ greatly in length.
- Semantic embeddings such as PaLM 2 Gecko collapse on character-level typos, while MinHash collapses on character-level augmentations.
Limitations
- Embedding speed is slower than MinHash+LSH, and the implementation is Python and not fully optimized.
- Near-Dup embeddings underperform on pairs with large length ratios, where Partial-Dup and MinHash do better.
- Chinese and Japanese results are slightly better for semantic embeddings, which the authors attribute to larger model capacity.
Experimental setup
Datasets: mC4 Wiki-40B NEWS-COPY CORE Near-Duplicates
Benchmarks: W4NT3D NEWS-COPY CORE Near-Duplicates Wiki-40B deduplication Gmail spam campaigns
Metrics: Recall@1 Adjusted Rand Index precision recall macro F1 accuracy V-Measure homogeneity completeness embedding speed
Baselines: LaBSE Multilingual USE Multilingual E5-Base PaLM 2 Gecko MinHash SimHash S-BERT
Builds on: RETVec Multi-Similarity Loss MinHash Gated Attention Unit
Closest papers by meaning
- MMTEB: Massive Multilingual Text Embedding Benchmark (ICLR 2025 · similarity 0.83)
- Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe (NeurIPS 2024 · similarity 0.82)
- Statistical Model-driven Similarity Hashing: Bridging Modalities for Efficient Unsupervised Retrieval (AAAI 2025 · similarity 0.81)
- On Affine Homotopy between Language Encoders (NeurIPS 2024 · similarity 0.81)
- Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers (ICLR 2025 · similarity 0.81)
- Very Efficient Listwise Multimodal Reranking for Long Documents (ICML 2026 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.