Mohamed bin Zayed University of Artificial Intelligence researchers have developed a benchmark exposing a significant weakness in leading artificial intelligence systems: models that recognise Arab cultural norms often struggle to communicate naturally in the dialects where those norms are expressed.The Abu Dhabi-based university’s ArabCulture-Dialogue project evaluates large language models across Modern Standard Arabic and 13 national dialects. It tests whether systems can interpret culturally grounded conversations, translate between standard Arabic and local speech, and generate responses in a requested dialect. The research was presented at the Association for Computational Linguistics conference in 2026.
The benchmark contains 6,942 dialogues, divided equally between Modern Standard Arabic and dialect versions. The conversations span 12 areas of everyday life and 54 more specific subjects, including weddings, food, parenting, agriculture, arts and games. Together they contain more than 343,000 words.
Researchers recruited 26 native Arabic speakers from 13 countries, with two participants representing each country. They revised the conversations, adapted them into natural local dialects and checked one another’s work for cultural and linguistic accuracy. Annotators were instructed to reproduce the way people actually speak rather than translate Modern Standard Arabic word for word.
The results reveal a distinction between recognising cultural behaviour and generating authentic dialectal language. GPT-5 and Gemini 2.5 Pro achieved accuracy of roughly 94 to 95 per cent on the multiple-choice cultural reasoning test. Performance deteriorated when models had to translate into dialect or continue conversations using a specified local variety.
Smaller open and Arabic-focused models showed considerably wider gaps. Hala-9B and SILMA-9B recorded scores ranging from the high 70s into the low 80s on parts of the cultural reasoning assessment, while some smaller systems operated near the benchmark’s 33 per cent random-choice level. ALLaM-7B scored 41.8 per cent on Modern Standard Arabic material and 39.8 per cent on dialect material in one evaluation setting, while Qwen3-8B recorded 35.4 per cent.
Country-specific customs presented a greater challenge than cultural practices shared across the Arab world. North African material proved particularly difficult for open models, while Emirati conversations were also identified among demanding dialect settings. Providing models with explicit information about the country and region sometimes improved performance, indicating that cultural knowledge may be encoded within the systems but is not consistently activated by conversational context alone.
The findings address a longstanding problem in Arabic-language AI development. Modern Standard Arabic dominates formal writing, education, broadcasting and many existing language datasets, but everyday communication relies heavily on dialects that vary sharply across the Gulf, Levant, North Africa and Nile Valley. Differences extend beyond vocabulary to grammar, pronunciation, idioms, politeness conventions and culturally specific ways of expressing intent.
That disparity has allowed some language models to appear highly capable in Arabic benchmarks while offering a less reliable experience when users address them in ordinary speech. Arabic is spoken by more than 400 million people, yet the availability of high-quality digital material remains uneven compared with English and other languages heavily represented in AI training data.
ArabCulture-Dialogue builds on an earlier dataset containing thousands of cultural commonsense questions covering 13 countries. The new work transforms those isolated questions into multi-turn conversations, forcing systems to track cultural meaning across an exchange instead of selecting answers to standalone prompts.
Quality controls were designed to prevent models from exploiting superficial clues. Researchers found during development that correct answers could sometimes be distinguished because they were longer or used distinctive opening phrases. Annotators therefore rewrote response choices to make their length, tone and structure comparable, requiring models to rely more heavily on cultural understanding.
The project forms part of a broader push to strengthen Arabic AI capabilities as the UAE expands investment in language models, computing infrastructure and artificial intelligence research. Work by regional institutions has produced Arabic-focused models and increasingly specialised benchmarks for dialect comprehension, cultural reasoning, speech recognition and multimodal systems.
ArabCulture-Dialogue still covers only 13 of the 22 Arab countries, and researchers acknowledge substantial linguistic differences can exist within individual countries. Dialect spelling is also far less standardised than Modern Standard Arabic, creating additional difficulties for both training and evaluation.
Topics
Live News