Research map / Large language models / Language, safety and interpretability
What Language is This? Ask Your Tokenizer.
Clara Meister, Ahmetcan Yavuz, Pietro Lesci, Tiago Pimentel
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use UniLID instead of fastText for dialect identification | LID · DSL-ML 2024 | macro F1: 0.532 -> 0.723 | Improved |
| use UniLID instead of fastText with 5 samples per language | LID trained on WiLI subsets · WiLI | accuracy: ~70% vs fails to generalize | Improved |
| use UniLID instead of fastText on out-of-domain short text | LID trained on WiLI · Tatoeba | macro F1: 0.160 -> 0.414 | Improved |
| use UniLID instead of fastText on formal legal text | LID trained on WiLI · UDHR (142 langs) | macro F1: 0.849 -> 0.868 | No clear effect |
| use UniLID instead of fastText on large label set | LID trained on GlotLID-C · GlotLID-C test | macro F1: 0.944 -> 0.929 | Hurt |
| use UniLID instead of fastText on large label set | LID trained on GlotLID-C · GlotLID-C test | macro FPR: 2.71e-5 -> 2.03e-5 | Improved |
| use UniLID instead of fastText for short inputs | LID on WiLI · WiLI 101-150 chars | accuracy (%): 90.73 -> 93.10 | Improved |
| use UniLID instead of fastText for long inputs | LID on WiLI · WiLI 1000+ chars | accuracy (%): 96.30 -> 96.53 | No clear effect |
| use pretrained Mistral-Nemo tokenizer vocabulary instead of dedicated tokenizer (ablation) | UniLID · GlotLID-C test | macro F1: 0.929 -> 0.912 | Hurt |
| use Qwen3 tokenizer vocabulary instead of dedicated tokenizer (ablation) | UniLID · GlotLID-C test | macro F1: 0.929 -> 0.904 | Hurt |
| increase vocabulary size (ablation) | UniLID · LID benchmarks | accuracy | No clear effect |
| add 5-10% character-level corruption noise (ablation) | UniLID vs fastText · WiLI | accuracy | No clear effect |
Closest papers by meaning
- GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languages (NeurIPS 2024 · similarity 0.83)
- The Foundations of Tokenization: Statistical and Computational Concerns (ICLR 2025 · similarity 0.83)
- Agnostic Language Identification and Generation (NeurIPS 2026 · similarity 0.83)
- Data Mixture Inference Attack: BPE Tokenizers Reveal Training Data Compositions (NeurIPS 2024 · similarity 0.83)
- Language Models over Canonical Byte-Pair Encodings (ICML 2025 · similarity 0.82)
- Enhancing Multilingual LLM Pretraining with Model-Based Data Selection (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
