Research map / Large language models / Language, safety and interpretability
CultureLLM: Incorporating Cultural Differences into Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CultureLLM, which fine-tunes culture-specific and unified LLMs on 50 World Values Survey seed questions expanded by an LLM-based semantic data augmentation with BERT-similarity filtering, outperforming GPT-3.5 and Gemini Pro on 59 culture-related classification datasets.
Problem
LLMs show cultural bias toward Western culture, and existing fixes (prompt engineering or culture-specific pre-training) are either ineffective for low-resource cultures or too costly in data and compute.
Approach
Use a small set of WVS survey questions as seed data, augment them with semantics-preserving rewrites (GPT-4 templates filtered by BERT cosine similarity, then context-aware synonym replacement), and fine-tune a GPT-3.5 model per culture or one unified model using each culture's WVS answers as labels.
Key results
- CultureLLM fine-tuned on GPT-3.5 outperforms GPT-3.5 by 8.1% and Gemini Pro by 9.5% on average F1 across 59 test sets, comparable to or exceeding GPT-4.
- CultureLLM-Llama-70b outperforms the vanilla Llama-2-70b-chat by 2.17% on average.
- Human and LLM raters judged generated samples semantically similar to seeds with an average score of 4.84/5 (human 4.60), with similarity passing 96.5%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune GPT-3.5 using 50 WVS seeds and semantically augmented data | GPT-3.5 · 59 culture-related datasets | average F1 score: +8.1% | Helped |
| Fine-tune GPT-3.5 using 50 WVS seeds and semantically augmented data | GPT-3.5 · 59 culture-related datasets | average F1 score: +9.5% | Helped |
| Fine-tune GPT-3.5 using 50 WVS seeds and semantically augmented data | GPT-3.5 · 59 culture-related datasets | average F1 score: +7.94% | Helped |
| Use one unified model across cultures instead of culture-specific models | CultureLLM · 59 culture-related datasets | average performance | Hurt |
| Use CultureLLM fine-tuning for open-ended generation | GPT-3.5 · Open-ended generation, Turkish culture | WinRate: -.062 | Hurt |
| Add semantic-template generation and synonym replacement to WVS seeds (ablation) | CultureLLM · Culture-related datasets | average performance | Helped |
| Fine-tune using only the 50 WVS seed samples (ablation) | CultureLLM · Culture-related datasets | performance | Mixed |
| Add semantic-template generation to WVS seeds, without synonym replacement (ablation) | CultureLLM · Culture-related datasets | performance | Helped |
| Increase generated training samples beyond 500 (ablation) | CultureLLM · Culture-related downstream tasks | performance | Hurt |
| Fine-tune in English rather than in local languages | GPT-3.5 · Culture-related tasks | performance | Helped |
| Fine-tune GPT-3.5 using culture-specific data | GPT-3.5 · Korean culture tasks | performance | No clear effect |
| Fine-tune Llama2-70b-chat with the CultureLLM approach | Llama2-70b-chat · Culture-related datasets | average performance: +2.17% | Helped |
Surprising findings
- Fine-tuning on English WVS-derived data outperformed fine-tuning on translated local-language data.
- Adding more than 500 generated samples degraded performance across tasks, suggesting overfitting or mode collapse.
Limitations
- Not implemented on large-scale open-source models; only a single Llama-2-70b experiment.
- Only classification tasks evaluated automatically; open-ended generation evaluated on 65 questions with a Gemini Pro judge.
- Sample diversity limited to sentence and word level.
Experimental setup
Datasets: World Values Survey (WVS-7) OffensEval2020 OSACT4 OSACT5 Multi-Platform TRAC2020 BAD Hate Speech (Bengali) BACD CCS CDial-Bias CValues
Benchmarks: OffensEval2020 OSACT4 OSACT5 Multi-Platform TRAC2020 BAD CValues MLMA HOF GermEval2018 HASOC2020 K-MHaS
Metrics: macro F1 accuracy (CValues) WinRate semantic similarity score (1-5)
Baselines: GPT-3.5 GPT-4 Gemini Pro RAG SeaLLM TaiwanLLM CultureBank Llama-2-70b-chat
Builds on: World Values Survey GPT-3.5 fine-tuning API retrieval-augmented generation
Closest papers by meaning
- CulturePark: Boosting Cross-cultural Understanding in Large Language Models (NeurIPS 2024 · similarity 0.90)
- Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages (ICLR 2025 · similarity 0.88)
- BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages (NeurIPS 2024 · similarity 0.88)
- Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.88)
- Attributing Culture-Conditioned Generations to Pretraining Corpora (ICLR 2025 · similarity 0.88)
- Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.