Content knowledge

E-EVAL is a comprehensive evaluation benchmark specifically designed for Chinese K-12 education, consisting of 4,351 multiple-choice questions across primary, middle, and high school levels covering nine subjects (Chinese, English, Politics, History, Ethics, Physics, Chemistry, Mathematics, Geography) to assess LLM capabilities in the Chinese K-12 education domain.

EduEval is a comprehensive hierarchical benchmark for evaluating LLMs in Chinese K-12 education, comprising 24 task types with over 11,000 questions organized across six cognitive dimensions (Memorization, Understanding, Application, Reasoning, Creativity, and Ethics) based on Bloom's Taxonomy and Webb's Depth of Knowledge. The benchmark uses authentic educational materials including real exam questions, classroom dialogues, student essays, and expert-designed prompts spanning primary through high school levels.

FoundationalASSIST introduces a 1.7-million interaction K-12 educational dataset with full question text, student responses, and Common Core alignment, specifically designed to evaluate whether LLMs can perform knowledge tracing (predicting student performance) and pedagogical grounding (understanding assessment item properties). The paper evaluates four frontier LLMs on these tasks, revealing significant gaps in their ability to predict student performance and understand item discrimination.

EduAdapt introduces a benchmark dataset of nearly 48k grade-labeled QA pairs across grades 1-12 and nine science subjects to evaluate whether LLMs can adapt their responses to different grade levels. The paper evaluates multiple open-source LLMs and finds they struggle to generate developmentally appropriate responses, especially for early-grade students.

FEANEL is a benchmark for evaluating LLMs' ability to provide fine-grained error analysis and pedagogical feedback on K-12 English writing, comprising 1,000 student essays with expert-annotated errors categorized by type, severity, and explanations. The benchmark specifically assesses whether AI systems can identify writing errors and provide educationally meaningful, interpretable feedback to support student learning.

This paper presents LearnLM-Tutor, a fine-tuned Gemini model for educational use, and introduces a comprehensive evaluation framework spanning seven diverse benchmarks (quantitative, qualitative, automatic, and human evaluations) grounded in learning science principles to assess pedagogical quality in K-12 AI tutoring systems. The work includes real-world deployment at Arizona State University and systematic evaluation of pedagogical dimensions including Socratic dialogue, adaptive scaffolding, and learning-centered interactions.

MDK12-Bench is a large-scale benchmark built from 141K real-world K-12 exam questions across six disciplines (Math, Physics, Chemistry, Biology, Geography, Information Science) with 6,225 structured knowledge points, designed to evaluate multimodal large language models on problem-solving capabilities across difficulty levels, temporal shifts, contextual shifts, and knowledge-driven reasoning. The benchmark includes a dynamic evaluation framework and knowledge-point reference-augmented generation (KP-RAG) to assess model generalization and the role of knowledge in problem-solving.

TutorBench is a purpose-built benchmark with 1,490 expert-curated samples evaluating LLMs on three core tutoring skills: generating adaptive explanations, providing actionable feedback, and creating effective hints for high-school and AP-level content. The benchmark uses sample-specific rubrics and LLM-judge evaluation to assess 16 frontier models, finding none exceed 56% overall performance and all achieve less than 60% pass rate on criteria related to guiding, diagnosing, and supporting students.

FACET is a teacher-facing LLM-based multi-agent system that generates personalized mathematics worksheets for grade 8 students by modeling learner profiles (cognitive proficiency and intrinsic motivation), adapting content through a teacher agent, and evaluating quality through an automated evaluator agent. The system was evaluated through automated agent-based assessment and exploratory feedback from K-12 in-service teachers on authentic curriculum content.

This paper introduces the AMMORE dataset of 53,000 middle-school math open-response answers from students in Africa using the Rori WhatsApp AI tutor, and evaluates LLM-based grading approaches (including chain-of-thought prompting) to improve automated scoring accuracy from 98.7% to 99.9%, demonstrating consequential validity through impacts on Bayesian Knowledge Tracing estimates of student mastery.

This paper evaluates LLM-based and statistical NLP methods for automated scoring of nationwide high school graduation essay exams in Estonia, comparing machine-generated scores against human raters across multiple rubric dimensions including content, argumentation, and language quality. The study demonstrates that automated scoring achieves reliability comparable to human raters while also examining bias, prompt injection risks, and providing personalized feedback capabilities.

MathDial presents a dataset of 3,000 one-to-one math tutoring dialogues where human teachers guide LLM-simulated students through multi-step reasoning problems using scaffolding questions and pedagogical moves, with extensive annotations for training and evaluating AI tutoring systems. The paper demonstrates that current LLMs fail at effective tutoring by revealing solutions too early or providing incorrect feedback, and shows how models finetuned on MathDial improve interactive tutoring performance.

This paper develops Bridge, a method that translates expert tutors' decision-making processes into a framework for LLMs to remediate elementary math mistakes, using a dataset of 700 real tutoring conversations with 1st-5th grade students. The work evaluates GPT-4 and Llama-2-70b on their ability to provide pedagogically sound responses to student errors when guided by expert decision models.

This paper presents MWPTutor, an LLM-based intelligent tutoring system for math word problems that combines structured pedagogical strategies (finite state transducers) with LLM flexibility, and evaluates it against GPT-4 through human evaluation studies. The system implements guardrails to prevent common tutoring pitfalls like answer-leaking while maintaining pedagogical control through predefined teaching strategies.

MathVC is an LLM-simulated multi-persona virtual classroom that creates AI-powered peer agents to facilitate collaborative mathematical problem-solving for middle school students. The system was evaluated with 14 U.S. middle-schoolers to assess engagement, motivation, and collaborative learning through simulated peer interactions with intentionally injected misconceptions.

This paper presents Physics-STAR, an LLM-powered tutoring system for high school physics education, and evaluates it through a controlled experiment with 12 high school sophomores against traditional lectures and generic LLM tutoring. The system provides step-by-step guidance, reflective learning prompts, and personalized scaffolding to improve conceptual understanding and problem-solving skills in physics.

This paper evaluates the effectiveness of LLM-generated analogies for teaching scientific concepts (biology and physics) through controlled in-class tests with high school students and classroom field studies with teachers. The study measures student understanding, learning outcomes, potential over-reliance, and teacher satisfaction with LLM-generated analogies, culminating in the development of a practical system for teachers to generate and refine teaching analogies.

This quasi-experimental study evaluates the use of ChatGPT to teach financial education to 110 secondary school students, comparing learning outcomes between an experimental group using AI tools and a control group receiving traditional instruction. The study measures student performance across five dimensions of financial literacy including planning, analysis, behavior, expense management, and investment initiative.

This paper reports results from an exploratory RCT (N=165) evaluating LearnLM, a pedagogically fine-tuned AI tutor, in UK secondary school mathematics classrooms, where expert tutors supervised all AI-generated messages. Students receiving supervised AI tutoring performed at least as well as those with human tutors alone, with significantly better knowledge transfer to novel problems (66.2% vs 60.7% success rate).

This paper presents RockStartIT Tutor, a GPT-4-powered AI tutoring system designed for secondary school students learning programming and computational thinking, using a curriculum-constrained knowledge base and pedagogically controlled prompting. The system was pilot-evaluated with 13 students and teachers using the Technology Acceptance Model to assess its effectiveness in providing scaffolded, personalized support.

This case study evaluates ChatGPT as a substitute teacher for 11th-grade chemistry students in the UAE, comparing student engagement and learning outcomes across cognitive domains (knowledge, application, reasoning) between ChatGPT-taught and traditionally-taught control groups using Bloom's taxonomy. The study found that while ChatGPT showed some promise in knowledge recall and reasoning, the control group significantly outperformed the experimental group, with double the percentage of students achieving good/outstanding results.

EducationQ is a multi-agent dialogue framework that evaluates LLMs' teaching capabilities through simulated teacher-student interactions, testing 14 models across 1,498 questions spanning 13 disciplines and 10 difficulty levels. The framework incorporates formative assessment principles to measure pedagogical effectiveness including questioning strategies, adaptive feedback, and scaffolding behaviors.

ConvoLearn introduces a dataset of 1,250 constructivist tutor-student dialogues in middle school Earth Science, grounded in knowledge-building theory across six pedagogical dimensions (cognitive engagement, formative assessment, accountability, cultural responsiveness, metacognition, and power dynamics). The paper demonstrates that fine-tuning LLMs on this dataset shifts model behavior toward constructivist teaching strategies, with the fine-tuned Mistral-7B significantly outperforming base models and Claude Sonnet in teacher evaluations.

This paper evaluates ChatGPT's ability to solve decimal math problems, assess correctness of student self-explanations, and generate feedback within the Decimal Point learning game for middle school students, using over 5,000 real student responses. The study assesses ChatGPT's content knowledge in decimals, automated grading accuracy (75%), and pedagogical quality of generated feedback using a structured rubric.

This paper presents a framework for finetuning LLMs to generate grade-appropriate educational content across six grade levels (lower elementary through adult), evaluating the pedagogical quality and age-appropriateness of AI-generated explanations using readability metrics and human evaluation with 208 participants.

This paper trains an open-source LLM (Llama 3.1 8B) to generate tutor utterances that maximize student learning outcomes in math tutoring dialogues by optimizing for both student response correctness and pedagogical quality using direct preference optimization. The approach uses a student model to predict correctness and GPT-4o to evaluate adherence to pedagogical principles, directly measuring the impact on student learning through dialogue interactions.

This paper develops and evaluates a dialogue-based intelligent tutoring system (ITS) for teaching sixth-grade students multiplication and division of fractions, using diagnostic teaching methodology with real-time error identification and adaptive instructional strategies. The system was tested through a quasi-experimental study with 66 sixth graders, showing significant learning gains compared to conventional classroom instruction.

This paper evaluates LLMs' mathematical reasoning by testing their ability to simulate novice learners making misconception-based errors and expert tutors diagnosing those misconceptions in grade-school math problems. The evaluation uses a dataset from Eedi's platform with multiple-choice questions tagged with specific mathematical misconceptions.

This paper introduces PACE, a personalized conversational tutoring agent for mathematics instruction that adapts teaching strategies based on students' learning styles using the Felder-Silverman model and employs Socratic teaching methods. The system is evaluated on its ability to identify individual learning styles and deliver personalized mathematics instruction through adaptive dialogue.

This paper evaluates ArtiBos, a gamified adaptive intelligent tutoring system designed for 9th-grade mathematics, examining its impact on students' problem-solving skills through system logs, student interviews, and teacher interviews in a Turkish high school setting.

This paper explores using GPT-4 to automatically contextualize math problems in CTAT (an intelligent tutoring system) to align with individual student interests at scale, aiming to increase engagement and learning outcomes. The authors use iterative prompt engineering to personalize problem contexts while preserving difficulty and pedagogical intent.

This study examines how conversational AI (LLM-based) supports K-12 mathematics educators during lesson preparation by analyzing educator-AI interactions across different educational contexts. The research evaluates AI's ability to adapt responses to educators' instructional needs, assessing accuracy, relevance, and usefulness of AI-generated pedagogical content and guidance.

This paper presents a randomized controlled trial evaluating GPT-4 as an interactive homework tutor for high-school ESL students, measuring impacts on learning outcomes (grammar gains) and student engagement compared to traditional static homework.

SPL (Socratic Playground for Learning) is a GPT-4-powered dialogue-based intelligent tutoring system that implements the Socratic teaching method to foster critical thinking and self-reflection through interactive questioning and personalized adaptive dialogues. The paper presents pilot experimental results from essay writing tasks demonstrating the system's potential to improve tutoring interactions in K-12 educational contexts.

This paper presents a randomized controlled trial (n=23) evaluating GPT-4's ability to personalize science texts for middle school students by profiling their learning preferences and rewriting content accordingly. The study measures student preference for personalized versus non-personalized texts, finding significant preference for aligned content.

AgentTutor is an LLM-powered multi-turn interactive tutoring system that uses generative agents to deliver personalized instruction by decomposing curricula, assessing learner cognitive levels based on Bloom's Taxonomy, dynamically adjusting teaching strategies, and providing reflective feedback through multi-turn dialogue. The system evaluates its effectiveness on benchmark datasets by measuring learning outcomes and teaching quality in interactive educational contexts.

This paper investigates whether LLMs' generation probabilities and error patterns align with student misconception patterns in multiple-choice questions, using real-world student response data across 3,202 MCQs. The study finds moderate correlation between LLM probabilities and student distractor selections, with LLMs frequently choosing the same incorrect answers that commonly mislead students.

This paper develops and evaluates a ChatGPT-based system that automatically generates personalized reading comprehension exercises (passages and multiple-choice questions) for middle school English learners in China, with extensive human evaluation by students, teachers, and native speakers demonstrating the quality meets or exceeds human-written materials.

HypoCompass is an LLM-powered intelligent tutoring system where students play teaching assistants who help debug LLM-generated buggy code, focusing on hypothesis construction skills in CS1 programming courses. The system was evaluated with 19 novices showing 12% improvement in debugging performance and 14% reduction in completion time.

This paper evaluates the quality and usefulness of educational questions generated by large language models (specifically GPT-3/InstructGPT) using controllable text generation with Bloom's taxonomy and difficulty levels, validated through teacher assessments across two domains. The study demonstrates that LLM-generated questions are judged by teachers as high quality and sufficiently useful for classroom use.

This paper develops and evaluates scaffolding techniques for multi-modal intelligent tutoring systems that guide children in describing images for language learning, using GPT-4V with pedagogical instructions grounded in four learning theories. The work constructs a seven-dimension rubric to evaluate the scaffolding process both manually and automatically.

ChatTutor is a full-fledged LLM-based intelligent tutoring system that provides course planning, instruction, and adaptive quizzes through three interconnected processes (interaction, reflection, reaction) with dynamic memory management. The system is evaluated through user studies and statistical analysis comparing it with real-world online learning platforms, measuring consistency, stability, and pedagogical effectiveness.

MalAlgoQA introduces a dataset of K-12 mathematics and reading comprehension questions (grades 3-11) with rationales for both correct answers and incorrect answer choices ('malgorithms'), designed to evaluate LLMs' counterfactual reasoning abilities by testing whether they can identify the flawed reasoning that leads to wrong answers. The paper evaluates multiple state-of-the-art LLMs and finds significant performance drops when identifying malgorithms versus correct algorithms, with important implications for AI tutoring systems that need to diagnose student misconceptions.

This paper compares three models (fine-tuned Mistral/GOAT, SBERT-Canberra, and GPT-4) for automatically scoring and providing qualitative feedback on middle-school students' open-ended math responses, evaluating both scoring accuracy and feedback quality using teacher judgments.

EduBench introduces a comprehensive benchmark dataset with 18,821 data points across 9 educational scenarios and 4,000+ educational contexts (covering K-12 and higher education subjects, different difficulty levels, and multiple question types), evaluated using 12 multi-dimensional metrics covering scenario adaptation, factual/reasoning accuracy, and pedagogical application. The benchmark assesses LLM capabilities in diverse educational tasks including assignment grading, study planning, tutoring, and psychological counseling, with both human and automated evaluation.

This empirical study evaluates the impact of Generative AI on elementary school students' writing skills in Indonesian language education, comparing two classes using Generative AI tools against two control classes in terms of completeness, structure, and creativity of written work. Results show significant improvement in writing abilities for students using Generative AI.

This study evaluates high school students' perspectives on ChatGPT's effectiveness in mathematics learning at SMA Negeri 6 Gorontalo through descriptive analysis of student questionnaires. The research examines student satisfaction, engagement, comprehension, and identifies barriers to ChatGPT adoption in K-12 mathematics education.

This quasi-experimental study evaluates an AI robot image recognition system where 7th-grade students teach the AI to recognize cell division images, comparing learning outcomes and motivation against traditional textbook instruction. The research directly measures the impact of this AI-based 'Learning by Teaching' approach on students' conceptual understanding of cell division and science learning motivation.

This paper introduces PBLBench, a benchmark for evaluating multimodal large language models (MLLMs) on their ability to assess STEM Project-Based Learning (PBL) outcomes, including evaluating research reports, code, experimental data, and videos using expert-validated criteria derived through the Analytic Hierarchy Process (AHP). The benchmark tests 15 MLLMs on grading and ranking PBL projects across multiple STEM disciplines with long-context multimodal inputs.

This paper presents DIRECT, a dialogue-based intelligent tutoring system for reading comprehension that asks questions, assesses student answers, provides hints, and engages in encouraging chat. The system is evaluated on a dataset constructed from RACE-M, which contains reading comprehension exercises from English exams for middle school students.