Research map / Large language models / Language, safety and interpretability
Enhancing Portuguese Variety Identification with Cross-Domain Approaches
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compiles PtBrVarId, a silver-labeled cross-domain corpus of European and Brazilian Portuguese, and fine-tunes BERTimbau with a two-step cross-domain protocol and delexicalization to build an open-source European vs. Brazilian Portuguese variety identifier.
Problem
Portuguese language variety identification (European vs. Brazilian) is hard because classifiers rely on named entities and topic rather than linguistic features and generalize poorly across domains; few open multi-domain resources exist.
Approach
Combine a multi-domain silver-labeled corpus with a cross-domain training protocol that tunes hyperparameters (including delexicalization probabilities for NER and POS masking) by validating on held-out domains, then fine-tune BERTimbau on the combined data.
Key results
- BERT model with delexicalization achieves 84.97% F1 on DSL-TL and 77.25% F1 on FRMT.
- Delexicalization improved FRMT F1 by about 13 points for N-gram and 10 points for BERT.
- N-gram model without delexicalization reaches only 46.95% F1 on the DSL-TL benchmark's FRMT entity bucket scenario, as reported in the paper's discussion of the entity bucket.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on a delexicalized corpus rather than a non-delexicalized corpus | N-gram model · FRMT | F1: approximately 13 percentage points | Helped |
| Train on a delexicalized corpus rather than a non-delexicalized corpus | BERT · FRMT | F1: approximately 10 percentage points | Helped |
| Use delexicalization during training | N-gram model · DSL-TL | F1 | Helped |
| Use delexicalization during training | BERT · DSL-TL | F1 | Helped |
| Use intermediate delexicalization levels (ablation) | N-gram and BERT models · PtBrVid validation set | F1 | Helped |
| Increase P_POS to high levels (ablation) | N-gram and BERT models · PtBrVid validation set | F1 | Hurt |
| Fine-tune BERT rather than use N-gram models | Portuguese LVI models · DSL-TL | F1 | Helped |
| Fine-tune BERT rather than use N-gram models | Portuguese LVI models · FRMT | F1 | Helped |
Surprising findings
- Intermediate delexicalization gave marginal gains, while high POS delexicalization hurt performance, contrary to prior reports of large drops from delexicalization.
- Models trained without delexicalization often assign the same label to both varieties of parallel FRMT sentence pairs, relying on entities.
Limitations
- Corpus is silver-labeled from metadata heuristics, with moderate annotator agreement (Fleiss' Kappa 0.57 for variety).
- Only two Portuguese varieties studied; African Portuguese not covered.
- Delexicalization applied only to training set; effect of delexicalizing validation set not studied.
Experimental setup
Datasets: PtBrVarId DSL-TL FRMT Europarl OSCAR CETEMPublico Hate-BR
Benchmarks: DSL-TL FRMT
Metrics: F1
Baselines: N-gram Naive Bayes
Builds on: BERTimbau BERT langid.py
Closest papers by meaning
- Tradutor: Building a Variety Specific Translation Model (AAAI 2025 · similarity 0.87)
- Trans-EnV: A Framework for Evaluating the Linguistic Robustness of LLMs Against English Varieties (NeurIPS 2025 · similarity 0.84)
- GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languages (NeurIPS 2024 · similarity 0.84)
- RedPajama: an Open Dataset for Training Large Language Models (NeurIPS 2024 · similarity 0.82)
- CultureLLM: Incorporating Cultural Differences into Large Language Models (NeurIPS 2024 · similarity 0.82)
- Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages (AAAI 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.