Research map / Large language models / Agents, code and math

Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Cultures

NeurIPS 2026 · Large language models › Agents, code and math · approach: Mathematical and structural reasoning benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds Global PIQA, a participatory, human-written commonsense reasoning benchmark with a culturally-specific non-parallel split (136 language varieties) and a parallel split (131 varieties), and evaluates frontier and open-weight LLMs on it.

Problem

Existing multilingual benchmarks are mostly translated from English and rarely capture culturally-specific everyday commonsense knowledge, especially for lower-resource languages.

Approach

A participatory construction model in which 350+ native-speaker researchers hand-write culturally-specific PIQA-style examples in their own languages (non-parallel split), complemented by a human-verified parallel split of culturally agnostic questions translated into 131 varieties to allow direct cross-lingual comparison.

Key results

• Gemma 4 31B, the best open-weight model, reaches 82.4% mean accuracy on the parallel split and 84.9% on the non-parallel split.
• Some proprietary systems exceed 90% accuracy averaged across languages on both splits.
• Even the best proprietary LLM shows up to a 68% accuracy gap between best and worst languages on the parallel split; Gemma 4 31B scores 60.5% on Sub-Saharan African languages vs 88.1% on European languages.

Surprising findings

• Lingala and Plateau Malagasy show large parallel-to-non-parallel drops (20% and 19%), suggesting weaker cultural knowledge relative to linguistic ability.
• Only 4.1% of the official non-parallel examples were written with LLM help, and 16 groups used LLMs for initial drafts, requiring heavy filtering (14.6% to 22.0% kept).

Limitations

• Only 100 examples per language in the official non-parallel split, limiting statistical power.
• Examples are snapshots by individual authors and may contain cultural stereotypes rather than representing whole cultures.
• Non-parallel split difficulty varies substantially across languages, making cross-language comparisons on it qualitative.

Experimental setup

Datasets: Global PIQA non-parallel split Global PIQA parallel split PIQA (English, inspiration) EWoK TRAM PROST HellaSwag (inspiration)

Benchmarks: Global PIQA non-parallel Global PIQA parallel

Metrics: accuracy mean accuracy across languages standard error

Baselines: Gemma 4 31B GPT-5.4 Claude Sonnet 4.6 Gemini 3.1 Pro Gemini 3 Flash Qwen 3.5 Llama 3.1 Aya

Builds on: PIQA HellaSwag Belebele Global MMLU

Closest papers by meaning

• BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages (NeurIPS 2024 · similarity 0.89)
• INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge (ICLR 2025 · similarity 0.89)
• Linguini: A benchmark for language-agnostic linguistic reasoning (NeurIPS 2025 · similarity 0.89)
• metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.89)
• PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts (NeurIPS 2025 · similarity 0.89)
• MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation (ICLR 2025 · similarity 0.88)