Research map / Large language models / Language, safety and interpretability
The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual and Monolingual Text Embedding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the Scandinavian Embedding Benchmark (SEB), a 24-task evaluation suite for Danish, Swedish, Norwegian (Bokmål/Nynorsk) and Bornholmsk text embeddings, integrated with MTEB, and benchmarks 26 models including commercial APIs.
Problem
Text embedding models for Scandinavian languages lack a reliable, comprehensive evaluation benchmark; MTEB has little non-English coverage, mostly translated datasets, and no naturally occurring multilingual retrieval data.
Approach
A Scandinavian-focused embedding benchmark built from naturally occurring datasets across many domains and task types (classification, bitext mining, clustering, retrieval), with a model registry documenting exact model prompts/implementations, ported into MTEB and limited to 2048 examples per dataset for laptop-feasible evaluation.
Key results
- Commercial embedding APIs lead: text-embedding-3-large averages 65.0 vs multilingual-e5-large 60.7 and e5-mistral-7b-instruct 60.4 across 24 tasks.
- The largest public-vs-commercial gap is in retrieval (over eight points); bitext mining is the only category where open models (SONAR, LaBSE) beat commercial APIs.
- Self-supervised SimCSE on dfm-encoder-large raises average from 41.4 to 46.6, while supervised multilingual e5 models outperform most pretrained encoders.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SimCSE self-supervised contrastive pretraining to dfm-encoder-large (ablation) | dfm-encoder-large · SEB, all tasks | Avg.: 41.4 -> 46.6 | Helped |
| add SimCSE self-supervised contrastive pretraining to dfm-encoder-large (ablation) | dfm-encoder-large · SEB, retrieval | Retr.: 20.1 -> 33.7 | Helped |
| use multilingual-e5-large instead of translate-e5-large (translate then embed) | SEB models · SEB, all tasks | Avg.: 47.7 -> 60.7 | Helped |
| use multilingual-e5-large instead of translate-e5-large (translate then embed) | SEB models · SEB, bitext mining | Bitext: 50.7 -> 60.1 | Helped |
| use commercial embedding APIs rather than publicly available embedding models | SEB embedding models · SEB, retrieval | Retr. | Helped |
| use open solutions rather than commercial APIs | SEB embedding models · SEB, bitext mining | Bitext | Helped |
Surprising findings
- Smaller monolingual Scandinavian encoders are competitive with much larger multilingual models on some tasks, contrary to expectations from scale.
- Pretrained encoders perform surprisingly well on linguistic acceptability and language identification tasks.
Limitations
- Limited domain coverage: legal, governmental, and medical domains only partly covered; little fiction, poetry, or religious text.
- Lacks pair classification and document reranking task categories.
- Relies on translated datasets (Massive Intent/Scenario) and cross-lingual transfer across Mainland Scandinavian languages.
Experimental setup
Datasets: DanFEVER LCC Language Identification Massive Intent Massive Scenario NoReC NorQuad Norwegian courts SNL Clustering SNL Retrieval ScaLA SweFAQ
Benchmarks: Scandinavian Embedding Benchmark (SEB)
Metrics: accuracy F1 V measure nDCG@10 average rank bootstrapped 95% confidence interval
Baselines: multilingual-e5-large e5-mistral-7b-instruct text-embedding-3-large text-embedding-3-small embed-multilingual-v3.0 LaBSE SONAR XLM-R large
Builds on: MTEB BEIR SentEval
Closest papers by meaning
- MMTEB: Massive Multilingual Text Embedding Benchmark (ICLR 2025 · similarity 0.95)
- SWEb: A Large Web Dataset for the Scandinavian Languages (ICLR 2025 · similarity 0.89)
- Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe (NeurIPS 2024 · similarity 0.89)
- DEPT: Decoupled Embeddings for Pre-training Language Models (ICLR 2025 · similarity 0.87)
- Analyzing Similarity Metrics for Data Selection for Language Model Pretraining (NeurIPS 2025 · similarity 0.86)
- EmbedLLM: Learning Compact Representations of Large Language Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.