Global PIQA
Free while signed in. Answers cite the passages they came from.
Global PIQA extends physical commonsense reasoning evaluation to 100+ languages and cultural contexts, revealing how language models handle everyday practical scenarios across diverse linguistic communities. The benchmark goes beyond translation to include culturally-contextualized scenarios, uncovering significant performance variations that challenge assumptions about universal physical understanding in AI systems. - **Multilingual physical reasoning at scale**: Rather than simple translations, Global PIQA provides culturally-adapted scenarios reflecting different environments and practices across 100+ languages. This enables assessment of whether models develop genuinely robust commonsense or merely memorize English-centric patterns about physical interactions. - **Cultural dependencies in "universal" concepts**: The research demonstrates measurable variations in how models reason about physical interactions depending on linguistic and cultural framing. This reveals that physical understandingâseemingly universalâexhibits language-specific dependencies in current AI systems trained primarily on English data. - **Performance gaps across languages**: Models show different proficiency levels when handling the same underlying physical reasoning concepts across languages. These variations expose potential biases in how systems generalize from English-dominant training data to other linguistic communities. - **Practical deployment implications**: The benchmark helps developers identify language-specific performance gaps before deploying models in non-English-speaking regions. This addresses a critical gap in multilingual AI evaluation for real-world applications requiring physical reasoning. - **Non-parallel evaluation design**: By creating context-aware adaptations rather than direct translations, Global PIQA more accurately captures how physical reasoning manifests in different cultural settings. This methodology provides more realistic assessment of model capabilities across global deployment scenarios.
Multilingual physical reasoning at scale: Rather than simple translations, Global PIQA provides culturally-adapted scenarios reflecting different environments and practices across 100+ languages. This enables assessment of whether models develop genuinely robust commonsense or merely memorize English-centric patterns about physical interactions.
Cultural dependencies in âuniversalâ concepts: The research demonstrates measurable variations in how models reason about physical interactions depending on linguistic and cultural framing. This reveals that physical understanding exhibits language-specific dependencies in current AI systems trained primarily on English data.
Performance gaps across languages: Models show different proficiency levels when handling the same underlying physical reasoning concepts across languages. These variations expose potential biases in how systems generalize from English-dominant training data to other linguistic communities.
Practical deployment implications: The benchmark helps developers identify language-specific performance gaps before deploying models in non-English-speaking regions. This addresses a critical gap in multilingual AI evaluation for real-world applications requiring physical reasoning.
Non-parallel evaluation design: By creating context-aware adaptations rather than direct translations, Global PIQA more accurately captures how physical reasoning manifests in different cultural settings. This methodology provides a more realistic assessment of model capabilities across global deployment scenarios.
Abstract
To date, there exist almost no culturally-specific evaluation benchmarks for large language models (LLMs) that cover a large number of languages and cultures. In this paper, we present Global PIQA, a participatory commonsense reasoning benchmark for over 100 languages, constructed by hand by 335 researchers from 65 countries around the world. The 116 language varieties in Global PIQA cover five continents, 14 language families, and 23 writing systems. In the non-parallel split of Global PIQA, over 50% of examples reference local foods, customs, traditions, or other culturally-specific elements. We find that state-of-the-art LLMs perform well on Global PIQA in aggregate, but they exhibit weaker performance in lower-resource languages (up to a 37% accuracy gap, despite random chance at 50%). Open models generally perform worse than proprietary models. Global PIQA highlights that in many languages and cultures, everyday knowledge remains an area for improvement, alongside more widely-discussed capabilities such as complex reasoning and expert knowledge. Beyond its uses for LLM evaluation, we hope that Global PIQA provides a glimpse into the wide diversity of cultures in which human language is embedded.
Get next weekâs papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack