Research map / Large language models / Retrieval and knowledge
Crafting Interpretable Embeddings for Language Neuroscience by Asking LLMs Questions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QA-Emb, interpretable text embeddings where each feature is a yes/no answer from an LLM to a selected question, and applies it to predicting fMRI voxel responses to narrative stories.
Problem
Text embeddings from LLMs are opaque, which hinders their use in scientific fields like language neuroscience where interpretability is the goal.
Approach
Build each embedding dimension as the answer to a yes/no question posed to a black-box autoregressive LLM; learning reduces to selecting a question set (via GPT-4-generated candidates plus elastic-net pruning) rather than updating model weights.
Key results
- QA-Emb achieves average fMRI test correlation 0.116, a 26% improvement over the Eng1000 interpretable baseline and slightly above BERT (5% better), but 7% below the best LLaMA model.
- QA-Emb with only 29 questions (0.122) outperforms the 985-feature Eng1000 (0.118) in fMRI prediction.
- A RoBERTa model distilled with 674 classification heads retains nearly full performance (0.113 vs 0.114 average correlation with probabilistic outputs).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use QA-Emb instead of Eng1000 | fMRI voxel response prediction · fMRI | test correlation: 26% average improvement | Helped |
| Use QA-Emb instead of BERT | fMRI voxel response prediction · fMRI | test correlation: 5% improvement | Helped |
| Use QA-Emb instead of the best cross-validated LLaMA model | fMRI voxel response prediction · fMRI | test correlation: 7% decrease | Hurt |
| Use only 29 selected questions instead of the full Eng1000 feature set (ablation) | QA-Emb fMRI prediction · fMRI | mean test correlation: 0.118 versus 0.122 | Helped |
| Distill question answering into a single RoBERTa model with binary heads (ablation) | QA-Emb · fMRI | mean test correlation: 0.114 -> 0.111 | No clear effect |
| Use predicted probabilities instead of binary answers in the distilled model (ablation) | QA-Emb distilled model · fMRI | mean test correlation: 0.111 -> 0.113 | No clear effect |
| Combine BM-25 with QA-Emb | MSMarco retrieval · MSMarco | Mean reciprocal rank: 0.77±.01 -> 0.80±.01 | Helped |
| Use QA-Emb alone instead of BM-25 | MSMarco retrieval · MSMarco | Mean reciprocal rank: 0.77±.01 -> 0.45±.01 | Hurt |
| Select task-relevant questions through zero-shot adaptation (ablation) | QA-Emb text clustering · Four text-classification datasets | average clustering score: 0.095 -> 0.191 | Helped |
| Select task-relevant questions through zero-shot adaptation (ablation) | QA-Emb text clustering · Rotten tomatoes, AG News, Emotion, Financial phrasebank | clustering score: 0.126±0.011 -> 0.248±0.016; 0.124±0.007 -> 0.166±0.012; 0.046±0.007 -> 0.057±0.010; 0.084±0.008 -> 0.292±0.017 | Helped |
Surprising findings
- Distilled RoBERTa with probabilistic outputs nearly matches the LLM-based QA-Emb despite answering only 88.5% of yes/no test answers matching the LLM.
Limitations
- Requires d LLM calls per embedding, which is computationally expensive.
- Performance depends on the LLM answering questions faithfully; accuracy varies across tasks (e.g., near chance on the math research question).
- Question selection is a heuristic, not a direct optimization over the discrete question space.
Experimental setup
Datasets: fMRI narrative story dataset (Huth lab podcasts, 3 subjects) MSMarco D3 binary classification collection (54 datasets) Financial phrasebank Emotion AG News Rotten tomatoes
Benchmarks: fMRI test correlation (3 subjects) D3 binary classification MSMarco information retrieval text clustering on Financial phrasebank, Emotion, AG News, Rotten tomatoes
Metrics: test correlation classification accuracy mean reciprocal rank Recall@1 Recall@5 clustering score
Baselines: Eng1000 BERT LLaMA-2 7B LLaMA-2 70B LLaMA-3 8B BM-25 bag of words bag of bigrams
Builds on: Eng1000 BERT LLaMA elastic net single-neuron explanation
Closest papers by meaning
- Far from the Shallow: Brain-Predictive Reasoning Embedding through Residual Disentanglement (NeurIPS 2025 · similarity 0.88)
- A General Framework for Producing Interpretable Semantic Text Embeddings (ICLR 2025 · similarity 0.87)
- Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction (NeurIPS 2024 · similarity 0.87)
- NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (ICLR 2025 · similarity 0.87)
- Identifying Query-Relevant Neurons in Large Language Models for Long-Form Texts (AAAI 2025 · similarity 0.86)
- Sparsity-Guided Holistic Explanation for LLMs with Interpretable Inference-Time Intervention (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.