Research map / Large language models / Language, safety and interpretability
Embedding Trust: Semantic Isotropy Predicts Nonfactuality in Long-Form Text Generation
Dhrupad Bhardwaj, Julia Kempe, Tim G. J. Rudner
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace LUQ with semantic isotropy (Gemini) | uncertainty scoring · Book Summaries, Phi 3.5 Mini | R^2: 0.33 -> 0.43 | Improved |
| replace LUQ with semantic isotropy (Mistral E5) | uncertainty scoring · FactScore-Bio, GPT 4.1 Mini | R^2: 0.12 -> 0.47 | Improved |
| replace LUQ with semantic isotropy (Nomic v1) on a stronger generator | uncertainty scoring · TriviaQA, GPT 4.1 Mini | R^2: 0.10 -> 0.14 | Mixed |
| replace LUQ with semantic isotropy (Gemini) | uncertainty scoring · TriviaQA, GPT 4.1 Mini | R^2: 0.10 -> 0.03 | Hurt |
| use Cohere v3.0 isotropy instead of LUQ under FactScore labels | uncertainty scoring · FactScore-Bio (FactScore), Llama 3.1 8B | R^2: 0.657 -> 0.75 | Improved |
| replace von Neumann entropy with Frobenius norm (ablation) | semantic isotropy score (Gemini) · TriviaQA, Phi 3.5 Mini | R^2: 0.43 -> 0.41 | No clear effect |
| replace von Neumann entropy with inverse trace (ablation) | semantic isotropy score (Gemini) · TriviaQA, Phi 3.5 Mini | R^2: 0.38 -> 0.43 | Improved |
| replace von Neumann entropy with LogDet (ablation) | semantic isotropy score (Nomic v1) · TriviaQA, Phi 3.5 Mini | R^2: 0.35 -> 0.39 | Hurt |
| use large instead of small OpenAI embedding (ablation) | semantic isotropy score · TriviaQA, Llama 3.1 8B | R^2: 0.16 -> 0.16 | No clear effect |
| use mean pooling instead of last-token embedding for Nomic v1 (ablation) | semantic isotropy score · pooling | R^2 | Improved |
| increase Qwen 3 embedding size from 0.6B to 8B (ablation) | semantic isotropy score · Book Summaries, Llama 3.1 8B | R^2: 0.18 -> 0.33 | Improved |
| increase Qwen 3 embedding size from 0.6B to 8B (ablation) | semantic isotropy score · TriviaQA, Llama 3.1 8B | R^2: 0.05 -> 0.00 | Hurt |
Closest papers by meaning
- MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Models (NeurIPS 2024 · similarity 0.88)
- To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contexts (ICLR 2025 · similarity 0.88)
- Precise Information Control in Long-Form Text Generation (NeurIPS 2025 · similarity 0.88)
- Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimension (ICML 2024 · similarity 0.88)
- Long-form factuality in large language models (NeurIPS 2024 · similarity 0.87)
- Distinguishing the Knowable from the Unknowable with Language Models (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
