Research map / Large language models / Language, safety and interpretability
Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Language Ranker, an intrinsic metric that ranks languages by the cosine similarity between LLM hidden representations of OPUS-100 non-English sentences and their English translations, and correlates it with pre-training data proportion and ARC/MMLU accuracy.
Problem
There is no quantitative method to evaluate LLM performance across low-resource languages, which are underserved relative to high-resource languages like English, German, and French.
Approach
Measure each language's representation similarity to English in intermediate layers of the LLM (averaged cosine similarity of last-token states over layers 5,10,15,20,25), rank languages by this score, and compare rankings across models via a longest common partial-order sublist ratio; add a double-variance (eigenvalue spread) measure of embedding-space distribution.
Key results
- High-resource languages such as German and French reach similarity around 0.72-0.74 with English on LLaMa2 7B, while low-resource languages like Kannada fall to about 0.24, with the paper reporting a strong correlation between similarity and pre-training data proportion.
- On MLMM ARC (4-shot, 100 samples per language), Gemma 7B reaches roughly 68-77% on high-resource languages versus 40-48% on low-resource Kannada, Armenian and Hindi.
- On Tatoeba pairs, High-High language pairs score higher than High-Low and Low-Low pairs for Gemma 7B, Mistral 7B and Qwen 7B, suggesting English is a reasonable baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use English-representation cosine similarity to rank languages | LLaMa2 7B · Languages with different pre-training corpus proportions | English similarity: German: 0.17%, 0.723; Kannada: ≤0.01%, 0.236 | Helped |
| Additional fine-tuning on Chinese corpus | Qwen · Chinese | English similarity | Helped |
| Evaluate high-resource rather than low-resource languages | LLaMa2 7B, Gemma 7B, Mistral 7B, and Qwen 7B · ARC and MMLU | accuracy: ARC: Chinese 27% vs Kannada 24% (LLaMa2 7B); 71% vs 48% (Gemma 7B); MMLU: Chinese 32% vs Kannada 21% (LLaMa2 7B); 54% vs | Helped |
| Increase language representation proportion in pre-training | LLaMa2 7B · Languages in the pre-training corpus | English similarity: German 0.17%, 0.723; Kannada ≤0.01%, 0.236 | Helped |
| Use larger model size (13B rather than 7B) | LLaMa2 and Qwen · High- versus low-resource languages | English similarity | No clear effect |
Surprising findings
- Qwen 7B's Chinese similarity rises in the last layers, which the authors attribute to Chinese-specific fine-tuning, while Qwen scores poorly on several European languages in MMLU/ARC.
- Qwen 7B shows lower ARC/MMLU accuracy on German and French than on Chinese despite similarity scores that suggest otherwise, indicating the metric does not always align with task accuracy.
Limitations
- Evaluation of reasoning uses only 100 random samples per language and 4-shot prompting, a small sample.
- Cosine similarity is acknowledged as not always reliable; the double-variance proxy is only approximately related to performance.
- Correlation with pre-training proportion is shown only for LLaMa2 using a small selection of languages from its technical report.
Experimental setup
Datasets: OPUS-100 Tatoeba-Challenge ARC MMLU
Benchmarks: ARC MMLU
Metrics: cosine similarity longest common partial order sublist ratio double variance accuracy
Baselines: MLMM-evaluation reasoning accuracy (ARC, MMLU)
Builds on: OPUS-100 Tatoeba-Challenge
Closest papers by meaning
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.91)
- BenchmarkCards: Standardized Documentation for Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.91)
- tinyBenchmarks: evaluating LLMs with fewer examples (ICML 2024 · similarity 0.91)
- LLMEval: A Preliminary Study on How to Evaluate Large Language Models (AAAI 2024 · similarity 0.90)
- Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.90)
- When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.