Research map / Large language models / Retrieval and knowledge

A General Framework for Producing Interpretable Semantic Text Embeddings

ICLR 2025 · Large language models › Retrieval and knowledge · approach: Domain-specific QA datasets · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CQG-MBQA, a framework that generates contrastive yes/no questions with an LLM over clustered text corpora and answers them with a cheap multi-task binary classifier, yielding interpretable binary text embeddings.

Problem

Black-box text embeddings are opaque, and existing question-based interpretable embeddings (QAEmb, ChiLL) rely on expert or prompt-engineered questions that are generic and costly to answer with LLMs at scale.

Approach

Contrastive Question Generation prompts an LLM with positive texts from a cluster and hard/easy negatives so questions answer yes only for the positives; a frozen encoder with multi-task BCE heads (MBQA) then answers thousands of questions cheaply, with post-hoc probing-based filtering and dedup.

Key results

• CQG-MBQA reaches 77.60 average Spearman on seven STS datasets, versus 65.20 for QAEmb-MBQA and 85.93 for AnglE.
• On retrieval (nDCG@10 average over seven datasets) CQG-MBQA scores 30.30, above QAEmb-MBQA (17.03) but below BM25 (47.97) and AnglE (60.00).
• Cognitive load on STS is 2.5-3.6x lower for CQG-MBQA (avg 449) than QAEmb-MBQA (avg 1,467); MBQA answers 96% of LLM answers correctly, and embedding MS MARCO dev costs about 41 USD versus 244,551 USD for GPT-4o-mini LLM-based QA.

Measured effects reported in the paper

Surprising findings

• Bag-of-Tokens BM25-like lexical baseline beats CQG-MBQA on retrieval for some datasets such as SciFact and NFCorpus is not the case, but BM25 outperforms CQG-MBQA on all retrieval datasets overall.
• QAEmb-MBQA is weaker than the Bag-of-Tokens baseline on several STS datasets.

Limitations

• Retrieval and clustering performance trails strong black-box models and BM25 on several datasets.
• Relies on an LLM to generate questions and on a teacher LLM (GPT-4o-mini) to label training data, so cost and LLM dependence remain.
• Cognitive load is proxied by the count of shared yes-dimensions, which is a coarse measure of interpretability and was not validated with human users.

Experimental setup

Datasets: MEDI2 MS MARCO NewsSpectrum

Benchmarks: SemEval STS 2012-2016 STS Benchmark SICK-R MS MARCO NewsSpectrum ArguAna FiQA-2018 NFCorpus SCIDOCS SciFact TwentyNewsgroupsClustering StackExchangeClusteringP2P

Metrics: Spearman correlation nDCG@10 V-Measure cognitive load accuracy macro F1 cost (USD)

Baselines: QAEmb-MBQA Bag-of-Tokens BM25 SBERT SimCSE AnglE OpenAI text-embedding-ada-002 WhitenedCSE GloVe USE BERT

Builds on: QAEmb ChiLL COGAM AnglE

Closest papers by meaning

• Crafting Interpretable Embeddings for Language Neuroscience by Asking LLMs Questions (NeurIPS 2024 · similarity 0.87)
• Question-Answering with Grammatically-Interpretable Representations (AAAI 2018 · similarity 0.86)
• Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe (NeurIPS 2024 · similarity 0.86)
• NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (ICLR 2025 · similarity 0.85)
• Rethinking Generative Large Language Model Evaluation for Semantic Comprehension (ICML 2024 · similarity 0.85)
• Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation (ICLR 2024 · similarity 0.85)