General reasoning

E-EVAL is a comprehensive evaluation benchmark specifically designed for Chinese K-12 education, consisting of 4,351 multiple-choice questions across primary, middle, and high school levels covering nine subjects (Chinese, English, Politics, History, Ethics, Physics, Chemistry, Mathematics, Geography) to assess LLM capabilities in the Chinese K-12 education domain.

EduEval is a comprehensive hierarchical benchmark for evaluating LLMs in Chinese K-12 education, comprising 24 task types with over 11,000 questions organized across six cognitive dimensions (Memorization, Understanding, Application, Reasoning, Creativity, and Ethics) based on Bloom's Taxonomy and Webb's Depth of Knowledge. The benchmark uses authentic educational materials including real exam questions, classroom dialogues, student essays, and expert-designed prompts spanning primary through high school levels.

FoundationalASSIST introduces a 1.7-million interaction K-12 educational dataset with full question text, student responses, and Common Core alignment, specifically designed to evaluate whether LLMs can perform knowledge tracing (predicting student performance) and pedagogical grounding (understanding assessment item properties). The paper evaluates four frontier LLMs on these tasks, revealing significant gaps in their ability to predict student performance and understand item discrimination.

EduAdapt introduces a benchmark dataset of nearly 48k grade-labeled QA pairs across grades 1-12 and nine science subjects to evaluate whether LLMs can adapt their responses to different grade levels. The paper evaluates multiple open-source LLMs and finds they struggle to generate developmentally appropriate responses, especially for early-grade students.

This paper presents LearnLM-Tutor, a fine-tuned Gemini model for educational use, and introduces a comprehensive evaluation framework spanning seven diverse benchmarks (quantitative, qualitative, automatic, and human evaluations) grounded in learning science principles to assess pedagogical quality in K-12 AI tutoring systems. The work includes real-world deployment at Arizona State University and systematic evaluation of pedagogical dimensions including Socratic dialogue, adaptive scaffolding, and learning-centered interactions.

MDK12-Bench is a large-scale benchmark built from 141K real-world K-12 exam questions across six disciplines (Math, Physics, Chemistry, Biology, Geography, Information Science) with 6,225 structured knowledge points, designed to evaluate multimodal large language models on problem-solving capabilities across difficulty levels, temporal shifts, contextual shifts, and knowledge-driven reasoning. The benchmark includes a dynamic evaluation framework and knowledge-point reference-augmented generation (KP-RAG) to assess model generalization and the role of knowledge in problem-solving.

This paper empirically evaluates LLM-based tutoring systems against traditional deep knowledge tracing (DKT) models for learner modelling in K-12 education, demonstrating that LLMs fall short in accurately tracking student knowledge over time even after fine-tuning. The study directly measures prediction accuracy, temporal coherence, and multi-skill mastery estimation using a large-scale K-12 dataset to assess whether LLMs can responsibly support adaptive instruction.

This paper presents a Socratic chatbot fine-tuned on open-source LLMs (Llama2 7B/13B) designed to foster critical thinking in students through structured questioning rather than providing direct answers. The system is evaluated through simulated student-chatbot interactions to assess its effectiveness in promoting reflection and critical thinking compared to standard chatbots.

This quasi-experimental study evaluates the use of ChatGPT to teach financial education to 110 secondary school students, comparing learning outcomes between an experimental group using AI tools and a control group receiving traditional instruction. The study measures student performance across five dimensions of financial literacy including planning, analysis, behavior, expense management, and investment initiative.

CODAE is a framework that fine-tunes open-source LLMs for AI tutoring by augmenting real student-tutor dialogues with Chain-of-Thought prompting to improve pedagogical quality. The paper addresses three key limitations (over-compliance, low response adaptivity, and threat vulnerability) and evaluates models on their ability to provide step-by-step guidance without prematurely revealing answers.

This paper proposes ERL4SIIP, an evolutionary reinforcement learning framework for AI tutoring that uses Socratic questioning to teach STEM interdisciplinary concepts, specifically designed to avoid cognitive offloading by promoting active knowledge construction rather than direct answer delivery. The system explicitly addresses the 'spoon-feeding trap' and evaluates pedagogical quality through measures of knowledge integration, transfer, and strategic diversity.

This paper evaluates LLMs' mathematical reasoning by testing their ability to simulate novice learners making misconception-based errors and expert tutors diagnosing those misconceptions in grade-school math problems. The evaluation uses a dataset from Eedi's platform with multiple-choice questions tagged with specific mathematical misconceptions.

This action research study evaluates the impact of ChatGPT-powered activities on critical thinking skills (using Bloom's taxonomy) across three university-level Chinese language classrooms (introductory L2, intermediate L2, and native speaker academic writing). The research examines which CT skills students apply when interacting with ChatGPT and how different task types influence CT skill application.

This quasi-experimental study evaluates the effect of AI chatbot-assisted problem-based learning on higher-order thinking skills (HOTS) in Grade 4 elementary students in Indonesia, comparing experimental and control groups using pre/post-tests and N-Gain scores.

This paper presents VATE (Virtual AI Teacher for Error Analysis), an LLM-based system deployed on the Squirrel AI platform that autonomously analyzes student mathematical errors from their work drafts, provides targeted error explanations, and engages in multi-round Socratic dialogue to guide correction. The system achieves 78.3% accuracy in error analysis for elementary mathematics and demonstrates improved student learning efficiency in real classroom deployment.

EduBench introduces a comprehensive benchmark dataset with 18,821 data points across 9 educational scenarios and 4,000+ educational contexts (covering K-12 and higher education subjects, different difficulty levels, and multiple question types), evaluated using 12 multi-dimensional metrics covering scenario adaptation, factual/reasoning accuracy, and pedagogical application. The benchmark assesses LLM capabilities in diverse educational tasks including assignment grading, study planning, tutoring, and psychological counseling, with both human and automated evaluation.

This paper compares symbolic, sub-symbolic, and neural-symbolic AI methods for predicting 7th-grade mathematics performance in Estonian primary school students using self-regulated learning data, evaluating generalizability and interpretability across balanced and imbalanced datasets. The study demonstrates that neural-symbolic AI better integrates cognitive, metacognitive, and motivational factors for more responsible and trustworthy educational data mining.

This study evaluates high school students' perspectives on ChatGPT's effectiveness in mathematics learning at SMA Negeri 6 Gorontalo through descriptive analysis of student questionnaires. The research examines student satisfaction, engagement, comprehension, and identifies barriers to ChatGPT adoption in K-12 mathematics education.

This quasi-experimental study evaluates an AI robot image recognition system where 7th-grade students teach the AI to recognize cell division images, comparing learning outcomes and motivation against traditional textbook instruction. The research directly measures the impact of this AI-based 'Learning by Teaching' approach on students' conceptual understanding of cell division and science learning motivation.

This paper develops and evaluates stepwise verification methods for detecting student reasoning errors in math problem-solving, showing that grounding tutor responses in explicit error detection improves feedback quality and reduces hallucinations in LLM-based dialog tutoring systems. The work collects a dataset of 1K annotated student solution chains and demonstrates that verifier-guided generation produces more targeted, correct responses compared to direct generation baselines.

TeachLM is a fine-tuned LLM for one-on-one tutoring, trained on 100,000 hours of authentic student-tutor interactions from Polygence. The paper introduces a novel multi-turn evaluation protocol using synthetic dialogues and demonstrates improvements in pedagogical interactions including doubled student talk time, better questioning strategies, and increased dialogue turns.

MDK12-Bench is a large-scale multimodal reasoning benchmark comprising 141K questions across six K-12 disciplines (math, physics, chemistry, biology, geography, information science) spanning grades 1-12, designed to evaluate MLLMs' reasoning capabilities using real-world K-12 examination questions with fine-grained knowledge annotations and difficulty labels.

This paper introduces STEM, a large-scale multimodal vision-language benchmark with 1,073,146 questions spanning 448 skills across science, technology, engineering, and math subjects, specifically designed based on K-12 curriculum (Pre-K through grade 8). The benchmark evaluates state-of-the-art AI models (GPT-3.5-Turbo, CLIP) on fundamental STEM skills, finding they perform well below elementary student levels (averaging 54.7%).

This paper develops and evaluates automated methods using large language models (GPT-4, GPT-4-mini, and ModernBERT) to assess critical thinking subskills in student-written argumentative essays, comparing zero-shot prompting, few-shot prompting, and supervised fine-tuning approaches against human coding. The work focuses on measuring higher-order reasoning skills including understanding/analyzing information, evaluating evidence, making inferences, and articulating arguments.

This study empirically evaluates the effectiveness of AI learning tools and customized curricula on improving critical thinking skills among 250 Indonesian K-12 students using quantitative SEM-PLS analysis. The research finds significant positive relationships between AI tool usage, curriculum customization, and students' critical thinking abilities.

This paper proposes Language model-based Knowledge Tracing (LKT), a novel framework that integrates pre-trained language models with knowledge tracing methods to predict student knowledge states and learning outcomes in online learning platforms. LKT leverages semantic information from question text to outperform traditional numerical-sequence-based knowledge tracing models and addresses the cold-start problem.

ES-KT-24 is a multimodal Knowledge Tracing benchmark dataset derived from educational game-playing videos covering K-12 Mathematics, English, Indonesian, and Malaysian subjects, containing 15,032 users and 7.7M interactions with synthetically generated question texts and game logs. The paper benchmarks deep learning-based KT models against language model-based approaches to predict student performance in game-based learning environments.

This survey paper systematically reviews LLM applications in education from a technology-centric perspective, covering student/teacher assistance, adaptive learning, and commercial tools, while organizing related datasets, benchmarks, and identifying deployment challenges. It provides a comprehensive taxonomy of how LLMs are being applied across multiple educational contexts including tutoring, assessment, content generation, and personalized learning.

This paper introduces MathQuest, a mathematics dataset derived from Indian 11th and 12th grade NCERT textbooks, and evaluates three large language models (LLaMA-2, WizardMath, MAmmoTH) on mathematical problem-solving tasks through fine-tuning experiments. The work benchmarks LLM performance on secondary-level mathematics content across varying complexity levels.

This paper investigates the 'Student Data Paradox' where training LLMs on student-tutor dialogue data to model student misconceptions inadvertently degrades the model's own factual knowledge and reasoning abilities. The authors propose hallucination tokens as a mitigation strategy to help LLMs distinguish between simulating student errors and providing accurate tutoring.

This paper introduces probabilistic models using Item Response Theory and novel discrete variational inference to predict whether students will answer mathematics exam questions correctly, releasing a large dataset of marked formal mathematics exam responses. The work focuses on adaptive learning systems that personalize question difficulty to maintain student engagement in the optimal learning zone.

This paper evaluates the performance of at least eight large language models (LLMs) on Chinese college entrance examination mathematics questions from 2019-2023, assessing accuracy, response time, logical reasoning, and creativity across different question types and difficulty levels.

This paper systematically benchmarks ChatGPT (GPT-3.5) and GPT-4 against human tutors across six programming education scenarios (program repair, hint generation, grading feedback, pair programming, contextualized explanation, and task synthesis) using introductory Python problems and real-world buggy student code. The evaluation uses expert annotations to assess pedagogical quality across scenarios relevant to digital tutoring, teaching assistance, and peer learning.

IndoMMLU is a multi-task language understanding benchmark for Indonesian that evaluates LLMs on 14,981 questions from primary school through university entrance exams across 64 tasks, including STEM, humanities, social science, Indonesian language proficiency, and nine local languages/cultures. The benchmark tests general reasoning, content knowledge, and multilingual capabilities of LLMs like GPT-3.5, BLOOMZ, and Falcon across Indonesian education levels.

This paper introduces the VNHSGE dataset, comprising over 19,000 multiple-choice questions and 300 essays across nine subjects from Vietnamese high school graduation examinations, designed to benchmark large language models' performance against student-level competencies. The authors evaluate ChatGPT and BingChat on this dataset, comparing their performance to Vietnamese students across subjects including mathematics, literature, sciences, and humanities.

CMMaTH is a Chinese multimodal mathematics benchmark containing 23,000 K-12 math problems (elementary to high school) designed to evaluate foundation models' ability to solve visual-mathematical problems across different grade levels and knowledge points. The paper introduces GradeGPT, an open-source evaluation tool for assessing model performance on this dataset.

This paper evaluates GPT-4o's multimodal and multilingual performance on physics concept inventories spanning multiple subjects and languages, comparing AI performance to undergraduate student benchmarks. The study assesses general reasoning, content knowledge mastery, and multimodal/multilingual capabilities using standardized physics assessments.

This paper explores using large language models (GPT-4 and Llama 2) to automatically generate and evaluate German reading comprehension multiple-choice test items, proposing a new evaluation metric called 'text informativity' based on answerability and guessability. The work evaluates both human and LLM-based assessment of generated items using zero-shot prompting.

This study evaluates the physics problem-solving capabilities of GPT-4o and o1-preview on German Physics Olympiad problems, comparing their performance against top high school students and analyzing solution quality beyond just answer correctness. The research examines implications for both summative and formative assessment in physics education, finding that LLMs can outperform human participants on advanced physics problems.

VisScience is a benchmark dataset of 3,000 K-12 multi-modal questions spanning mathematics, physics, and chemistry (elementary through high school), designed to evaluate multi-modal large language models' scientific reasoning capabilities across 21 subjects and 5 difficulty levels. The paper evaluates 25 MLLMs on this benchmark, finding closed-source models generally outperform open-source ones, with best accuracies of 53.4% (math), 38.2% (physics), and 47.0% (chemistry).

This paper proposes using Signal Temporal Logic (STL) to evaluate confidence trajectories in Chain-of-Thought reasoning for LLMs solving high school mathematics problems, aiming to improve calibration and reduce overconfident incorrect answers. The method is tested on Chinese Gaokao mathematics questions to provide more reliable uncertainty estimates for educational AI systems.

This paper presents ProgresSyn, an algorithm that synthesizes progressions of subtasks for block-based visual programming tasks (like Hour of Code and Karel), breaking complex problems into simpler steps. The work evaluates this approach both with AI agents (neural program synthesizers) and through a user study with novice K-12 programmers, demonstrating improved task completion rates.

This paper describes Brilla AI, an AI system designed to compete in Ghana's National Science and Maths Quiz (NSMQ), a live secondary school competition covering biology, chemistry, physics, and mathematics. The system was deployed to compete in the Riddles round of the 2023 NSMQ Grand Finale and the authors propose it as a benchmark for evaluating AI teaching assistants that could eventually provide one-on-one science tutoring to African students.

BanglaMATH is a benchmark dataset of 1,700 Bangla math word problems from grades 6-8, designed to evaluate LLMs' mathematical reasoning capabilities in a low-resource language. The paper assesses commercial and open-source LLMs on elementary-level math problems, examining performance across grade levels, reasoning complexity, and robustness to distracting information.

This paper introduces GSM1k, a new benchmark designed to mirror the style and complexity of GSM8k (grade school math problems) to evaluate whether LLMs genuinely reason or have memorized training data. The study finds evidence of dataset contamination and overfitting in several model families, with performance drops of up to 8% on the novel benchmark.

MathScale proposes a method to generate large-scale mathematical reasoning training data using GPT-3.5 by extracting concepts from seed questions and creating a concept graph, then uses this data to fine-tune open-source LLMs. The paper introduces MWPBENCH, a comprehensive benchmark covering K-12 through college-level math word problems across ten datasets including GSM8K and MATH.

GSM-PLUS is a robustness benchmark for evaluating large language models' mathematical reasoning capabilities by perturbing grade school math problems from GSM8K with various mathematical variations (numerical substitutions, problem understanding changes, distractor insertions, etc.). The benchmark tests whether LLMs truly understand mathematical concepts or rely on shortcuts, revealing accuracy gaps of up to 20% when questions are slightly modified.

CMATH is a dataset of 1,700 Chinese elementary school math word problems (grades 1-6) designed to benchmark LLMs' arithmetic and reasoning capabilities against human grade-level performance, evaluating models like GPT-4, ChatGPT, and others on their ability to solve age-appropriate math problems.

This paper explores ChatGPT's capability to solve physics calculation problems at the high school level, demonstrating its ability to provide step-by-step solutions, explain reasoning, and generate practice exercises. The study evaluates ChatGPT's performance on physics problems using prompting techniques like chain-of-thought reasoning.

This paper presents Orca-Math, a 7-billion parameter language model trained on 200K synthetic math problems that achieves 86.81% accuracy on GSM8K (a grade school math benchmark) without requiring ensemble methods, code execution, or external tools. The work uses an iterative learning approach where the model practices solving problems and receives feedback from a teacher model (GPT-4).