Research map / Vision-language and multimodal / CLIP and zero-shot

Repetition Improves Language Model Embeddings

ICLR 2025 · Vision-language and multimodal › CLIP and zero-shot · approach: Language models for zero-shot reasoning · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes echo embeddings, which repeat the input in a prompt and pool embeddings from the second copy, letting causal LMs produce strong text embeddings without architectural changes or fine-tuning.

Problem

Causal (autoregressive) LMs produce weak text embeddings because early tokens cannot attend to later tokens; prior fixes either require bidirectional attention modification with extra MLM training or large synthetic data.

Approach

Prompt the LM to repeat/rewrite the input so the second occurrence can attend to the whole input under causal attention; pool mean token embeddings from the second copy, enabling bidirectional-like information without changing the architecture.

Key results

• Zero-shot echo embeddings with Mistral-7B-Instruct reach 48.64 average MTEB (56 datasets) vs 42.38 for classical mean-pooled embeddings and 43.69 for PromptEOL.
• Fine-tuned compute-matched echo embeddings reach 64.66 MTEB vs 63.98 for classical causal and 64.23 for classical with bidirectional attention; LLM2Vec-supervised reports 64.80.
• Casting Mistral-7B to bidirectional attention without training yields 58.24 on MTEB-MINI vs 59.78 for echo, while LLaMA-2-7B classical causal scores 47.27 vs 56.99 for echo.

Measured effects reported in the paper

Surprising findings

• Casting Mistral-7B to bidirectional attention without further training does not hurt performance, unlike LLaMA-2-7B and S-LLaMA-1.3B.
• Compute-matched echo embeddings (halved input length) slightly outperform full-compute echo embeddings for Mistral-7B in zero-shot.

Limitations

• Echo embeddings roughly double inference compute unless input is shortened, and training compute also doubles.
• For very small token budgets (under about 64 tokens zero-shot), bidirectional classical embeddings outperform echo embeddings.
• Evaluation on a 28-dataset MTEB subset for many ablations; full MTEB only for main tables.

Experimental setup

Datasets: MTEB English (56 datasets) MTEB-MINI (28 datasets) Publicly available embedding training datasets GPT-4-generated toy dataset STSBenchmark

Benchmarks: MTEB MTEB-MINI

Metrics: average MTEB score accuracy on toy dataset

Baselines: Classical embeddings PromptEOL BERT RoBERTa LLM2Vec-unsupervised LLM2Vec-supervised GritLM-7B Classical + bidirectional attention

Builds on: LLM2Vec PromptEOL SimCSE GradCache Sentence-BERT

Closest papers by meaning

• Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe (NeurIPS 2024 · similarity 0.88)
• EmbedLLM: Learning Compact Representations of Large Language Models (ICLR 2025 · similarity 0.87)
• MMTEB: Massive Multilingual Text Embedding Benchmark (ICLR 2025 · similarity 0.86)
• NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (ICLR 2025 · similarity 0.86)
• The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual and Monolingual Text Embedding (NeurIPS 2024 · similarity 0.86)
• SimReg: Achieving Higher Performance in the Pretraining via Embedding Similarity Regularization (NeurIPS 2026 · similarity 0.86)