Multilingual capabilities

This paper evaluates the multilingual performance of large language models (GPT-4o, Gemini, Claude, Llama, Mistral) on four K-12 educational tasks: identifying student misconceptions, providing targeted feedback, interactive tutoring, and grading translations across nine languages including English, Mandarin, Hindi, Arabic, German, Farsi, Telugu, Ukrainian, and Czech.

This empirical study evaluates the impact of Generative AI on elementary school students' writing skills in Indonesian language education, comparing two classes using Generative AI tools against two control classes in terms of completeness, structure, and creativity of written work. Results show significant improvement in writing abilities for students using Generative AI.

This paper presents an automated multilingual pipeline for generating, solving, and evaluating K-10 math problems using three commercial LLMs (GPT-4o-mini, Gemini 2.5 Flash, Qwen-plus) across English, German, and Arabic. The study reveals consistent linguistic bias, with English solutions rated highest and Arabic lowest, highlighting equity concerns in AI-based educational tools.

This paper develops and evaluates automatic readability assessment models for science education documents in Basque, Spanish, and English for secondary education (ages 12-16), using both machine learning and deep learning approaches to help teachers select appropriately leveled texts for multilingual STEM instruction. The work creates three annotated corpora (BasqueARA, Agrega2Es, Agrega2En+) and tests feature-based and transformer-based models to classify text difficulty levels.

IndoMMLU is a multi-task language understanding benchmark for Indonesian that evaluates LLMs on 14,981 questions from primary school through university entrance exams across 64 tasks, including STEM, humanities, social science, Indonesian language proficiency, and nine local languages/cultures. The benchmark tests general reasoning, content knowledge, and multilingual capabilities of LLMs like GPT-3.5, BLOOMZ, and Falcon across Indonesian education levels.

This paper evaluates GPT-4o's multimodal and multilingual performance on physics concept inventories spanning multiple subjects and languages, comparing AI performance to undergraduate student benchmarks. The study assesses general reasoning, content knowledge mastery, and multimodal/multilingual capabilities using standardized physics assessments.

This paper presents a user study with 16 DHH (d/Deaf and Hard-of-Hearing) learners interacting with LLM-powered AI tutors that embody different personas based on varying levels of experience in DHH education. The study evaluates accessibility preferences, cultural knowledge, transparency needs, and multimodal support requirements (particularly sign language) for DHH learners using conversational AI tutors in online learning contexts.

EXAMS-V is a multilingual multimodal benchmark dataset containing 20,932 multiple-choice exam questions across 20 school disciplines in 11 languages, designed to evaluate vision-language models on their ability to reason over integrated text, images, tables, diagrams, and scientific symbols from real school exams.

This paper proposes an automated system to assess children's speech fluency in low-resource languages (Tamil and Malay) by combining a fine-tuned multilingual ASR model with GPT-based classification of objective fluency metrics (phonetic/word error rates, speech rate, pause patterns). The system is designed for mother tongue language learning in Singapore primary schools, addressing the challenge of automated oral assessment in educational contexts.

This paper evaluates ChatGPT's accuracy in solving 586 Korean mathematics questions (achieving 66.72% accuracy) and its ability to rate question difficulty using eleven criteria, examining the model's robustness in multilingual educational settings.

This paper develops and evaluates a PBL-centered English language course for Korean college students that integrates AI-based learning tools (Naver Papago, Google Translate, Grammarly) for translation and grammar checking. The study examines student learning experiences across cognitive, affective, linguistic, and knowledge construction dimensions, finding both positive outcomes and concerns about tool dependency and technical limitations.

This mixed-methods study evaluates Duolingo's game-based mobile application for improving English speaking proficiency among high school students in Turkey, using pre/post/retention tests and metaphorical perceptions. The research examines gamification strategies and their effectiveness in developing oral language skills in a K-12 setting.

KazMMLU is an MMLU-style benchmark dataset of 23,000 multiple-choice questions in Kazakh and Russian covering high school and university subjects (STEM, humanities, social sciences) with Kazakhstan-specific content, used to evaluate multilingual LLMs like Llama3.1, Qwen-2.5, GPT-4o, and DeepSeek V3.

TR-MMLU is a Turkish language benchmark comprising 6,200 multiple-choice questions across 62 sections drawn from Turkey's education system (including university entrance exams and open education faculty exams) to evaluate large language models' knowledge and linguistic capabilities in Turkish. The benchmark tests factual recall, conceptual understanding, logical reasoning, and cultural context across diverse academic disciplines.

This paper proposes a framework for controlling language difficulty in LLM-generated dialogue responses for second language (L2) learners by using linguistic features (readability, syntactic, lexical) and introduces Dilaprix, a metric to quantify dialogue complexity. The work focuses on adapting AI dialogue systems to match learners' proficiency levels, particularly for speaking practice in language learning.

This paper introduces a French question-answering dataset from Alloprof (a Quebec K-12 help website) containing 29,349 student questions across subjects, and presents a case study using BERT models for information retrieval to match student questions with relevant educational resources. The work focuses on creating and evaluating retrieval systems to help students find answers quickly on the platform.

This paper describes the EdUKate project, which uses machine translation to convert Czech digital learning materials (9,000+ interactive exercises) into Ukrainian, English, and German for primary and secondary school students. The project focuses on technical MT system development and evaluation for educational content translation, rather than measuring AI's impact on student learning outcomes.

This paper presents README++, a multilingual benchmark for evaluating readability assessment of sentences across 5 languages (Arabic, English, French, Hindi, Russian) and 112 data sources, focusing on second language learners using the CEFR scale. The work evaluates various language models' ability to predict text difficulty levels in supervised, unsupervised, and few-shot settings.

This paper investigates how language simplification (Normal, Simple, ELI5) affects LLM-generated definitions of homonyms, finding that simplified prompts drastically reduce word sense awareness and completeness. The authors propose a 'Helpful Sense Awareness' metric and demonstrate that fine-tuning improves homonym definition quality across multiple languages.

This paper investigates the impact of AI-powered NLP tools on English as a Foreign Language (EFL) instruction at a university in Benin using mixed-methods research, focusing on collaborative and interactive online learning in the post-pandemic context. The study examines teacher and student perceptions, learning methodologies, and the role of teacher ICT training in integrating AI-supported collaborative learning.

This is a survey paper reviewing multiple AI applications in language learning, including cyberbullying detection, AI-powered translation for foreign language teaching, English pronunciation evaluation using deep learning, and conversational agents. The paper compiles existing research on AI in language education but does not present original benchmarks, evaluation methodologies, or empirical studies measuring AI's impact on K-12 student learning outcomes.

INCLUDE is a multilingual language understanding benchmark comprising 197,243 QA pairs from local exam sources across 44 languages, designed to evaluate LLM performance in regional and cultural contexts. The benchmark draws from educational, professional, and practical tests from different countries to capture authentic regional knowledge rather than translated content.

This paper proposes a pedagogical model combining MOOCs with automatic sign language translation technology to provide deaf students access to educational content in their native sign language, demonstrating this through a digital literacy course for schoolteachers.

This paper discusses the introduction of digital technologies and AI tools (DeepL, Grammarly, chatbots) in foreign language teaching at the university level, focusing on developing students' linguo-digital competencies. It presents a theoretical overview and student survey results on using AI technologies in higher education language learning.

MuBench is a multilingual benchmark evaluating LLM capabilities across 61 languages, covering tasks like natural language understanding, commonsense reasoning, factual recall, and academic reasoning, with cross-lingual alignment to enable fair comparisons. The paper focuses on assessing multilingual LLM performance gaps and cross-lingual transfer dynamics through pretraining experiments.

This paper is a conceptual/theoretical discussion of how AI technologies (language learning platforms, translation tools, NLP) can support Arabic and English language learning, framed through the author's own 'AlAfnan Taxonomy' which extends Bloom's taxonomy to include cognitive, affective, and psychomotor domains. The paper does not present empirical evaluations, benchmarks, or systematic assessments of AI tools in educational settings.

This paper benchmarks 11 large language models on their performance when prompted in Simplified versus Traditional Chinese, focusing on regional term choice and hiring name choice tasks to identify biases between the two Chinese variants. The work examines representational harms and differential LLM responses across Chinese character sets but does not focus on K-12 education applications.

This paper presents a comprehensive evaluation of ChatGPT across 23 datasets covering 8 NLP tasks, testing multitask, multilingual, and multimodal capabilities, with focus on reasoning accuracy, hallucination problems, and interactive prompt engineering. The evaluation includes logical reasoning, commonsense reasoning, machine translation, summarization, and other general NLP tasks in multiple languages.

MEGA is a comprehensive benchmark evaluating generative LLMs (ChatGPT, GPT-4, BLOOMZ) on 16 NLP datasets across 70 languages, comparing their performance to state-of-the-art non-autoregressive models on tasks like question answering, natural language inference, and machine translation. The study focuses on multilingual capabilities and cross-lingual performance gaps, particularly for low-resource languages.

This paper introduces Cross-Lingual-Thought (XLT) prompting, a template-based method to improve large language models' performance across multiple languages by stimulating cross-lingual reasoning. The approach is evaluated on 7 NLP benchmarks covering reasoning, understanding, and generation tasks in high-resource and low-resource languages.

This paper introduces cross-lingual prompting (CLP) to improve zero-shot chain-of-thought reasoning across different languages in large language models, proposing methods to align representations between languages and enhance multilingual reasoning capabilities. The work evaluates performance on mathematical reasoning and other benchmarks across multiple languages.

This paper introduces EthioEmo, a multi-label emotion classification dataset for four Ethiopian languages, and evaluates various large language models (encoder-only, encoder-decoder, and decoder-only) on emotion understanding tasks in both low-resource and high-resource languages.

This paper analyzes how quantization (a compression technique for LLMs) affects performance across multiple languages, finding that automatic metrics underestimate degradation, non-Latin script languages are harmed more, and challenging tasks like mathematical reasoning degrade fastest. The study focuses on technical compression methods for multilingual LLMs rather than educational applications or K-12 contexts.

This paper evaluates multilingual large language models' ability to understand and reason with proverbs and sayings from six different languages within conversational contexts, constructing the MAPS benchmark dataset. The work focuses on measuring cultural reasoning gaps across languages and whether models can properly interpret figurative language in context.

This paper investigates how English-centric reasoning language models (RLMs) generalize to multilingual mathematical reasoning tasks through test-time compute scaling, examining language-mixing patterns and cross-domain generalization. The work focuses on technical aspects of multilingual AI reasoning capabilities rather than K-12 educational applications or learning outcomes.

This paper introduces EverydayMMQA, a framework for creating culturally-grounded multimodal datasets, and OASIS, a large-scale dataset with 14.8M QA pairs combining speech, images, and text in English and Arabic varieties across 18 countries. The work focuses on evaluating multimodal LLMs for everyday reasoning and cultural awareness in low-resource languages.

This paper benchmarks large language models (LLMs), primarily GPT-3.5-turbo, GPT-4, and OpenChat-3.5, on diverse Persian language tasks including classic NLP tasks, reasoning tasks, and knowledge-based tasks. The study introduces two new Persian benchmarks based on elementary school math questions and entrance exams for 7th and 10th grades, but focuses on evaluating general LLM capabilities in Persian rather than their pedagogical effectiveness or impact on student learning.

This paper analyzes user interactions with TALKA, a generative AI chatbot designed for Hakka language learning, examining 7,077 user utterances through Bloom's Taxonomy and dialogue act categorization to understand information-seeking behaviors and cognitive processes in low-resource language learning.

Open Brain AI is a computational platform that uses AI (machine learning, NLP, large language models, and speech-to-text) to automatically analyze multilingual spoken and written language productions, primarily for clinical assessment of neurogenic speech/language disorders and research purposes, with a secondary mention of potential classroom student performance evaluation.

This paper presents ERUPD, a parallel dataset of 75,146 English-Roman Urdu sentence pairs created through synthetic data generation and real-world messaging data, intended for machine translation and NLP tasks. The dataset aims to support multilingual education and cross-cultural exchange by bridging linguistic gaps for Roman Urdu speakers.

SwitchLingua is a large-scale multilingual code-switching dataset (420K text samples, 80 hours audio across 12 languages) designed to benchmark Automatic Speech Recognition (ASR) systems on code-switching scenarios, with a novel evaluation metric (SAER) that incorporates semantic information.

MM-EVAL is a multilingual meta-evaluation benchmark designed to assess LLM-based evaluators (LLM-as-a-judge and reward models) across 18-122 languages, testing whether evaluator LLMs can reliably assess multilingual outputs through pairwise accuracy and consistency metrics. The benchmark focuses on general multilingual evaluation challenges rather than educational contexts, with subsets covering chat, reasoning, safety, language hallucination, and linguistics.

This paper presents IndicQuest, a benchmark question-answering dataset for evaluating how well multilingual LLMs represent knowledge across 19 Indic languages and English, covering five India-specific domains. The dataset evaluates factual knowledge representation using reference-based metrics and LLM-as-a-judge evaluation.

This paper introduces KoLasSimpleQA, a multilingual benchmark for evaluating factual knowledge and hallucination in Large Language Models across 9 languages, covering both general domain and language-specific knowledge (history, culture, regional traditions). The benchmark uses simple fact-based questions with single knowledge points, objective answers, and temporal stability to assess LLMs' factual memory and self-awareness.

MENLO is a framework for evaluating native-like quality of LLM responses across 47 languages by breaking down quality into four dimensions (language quality, cultural/linguistic alignment, factual correctness, and writing style). The paper creates a dataset of 6,423 human-annotated preference pairs and develops LLM judges that can evaluate multilingual response quality.

This paper introduces NEU-ESC, a Vietnamese dataset for sentiment analysis and topic classification of university student forum comments, and benchmarks various BERT-based models on these tasks. The work focuses on social listening in higher education contexts using NLP techniques.

This paper explores using Large Language Models (LLMs) like ChatGPT and GPT-4 to generate synthetic training data for multilingual commonsense reasoning tasks (XCOPA, XWinograd, XStoryCloze), then evaluates whether fine-tuning smaller models on this synthetic data improves performance in low-resource language settings.

This paper presents M-RewardBench, a multilingual benchmark for evaluating reward models used in aligning large language models with human preferences across 23 languages, testing capabilities in chat, safety, reasoning, and translation tasks. The work focuses on technical evaluation of reward model performance across languages rather than K-12 educational applications.

OpenEval is a comprehensive evaluation platform for Chinese large language models (LLMs) that benchmarks them across capability (NLP tasks, knowledge, reasoning), alignment (bias, offensiveness), and safety (power-seeking, self-awareness risks). The platform evaluates general-purpose Chinese LLMs on various dimensions but does not specifically focus on K-12 education contexts or pedagogical applications.

CodeApex is a bilingual benchmark for evaluating Large Language Models on programming tasks including comprehension, code generation, and code correction, primarily using C++ problems and comparing various LLMs like GPT-4.