Pedagogical knowledge

This paper presents LearnLM-Tutor, a fine-tuned Gemini model for educational use, and introduces a comprehensive evaluation framework spanning seven diverse benchmarks (quantitative, qualitative, automatic, and human evaluations) grounded in learning science principles to assess pedagogical quality in K-12 AI tutoring systems. The work includes real-world deployment at Arizona State University and systematic evaluation of pedagogical dimensions including Socratic dialogue, adaptive scaffolding, and learning-centered interactions.

EduGuardBench is a dual-component benchmark designed to evaluate LLMs acting as simulated teachers, measuring both pedagogical fidelity (role-playing accuracy, teaching competence) and adversarial safety (resistance to jailbreaking, handling of academic misconduct requests). The benchmark identifies harmful teaching behaviors (incompetence, indolence, offensiveness) and uses persona-based adversarial prompts to test ethical boundaries specific to educational contexts.

FACET is a teacher-facing LLM-based multi-agent system that generates personalized mathematics worksheets for grade 8 students by modeling learner profiles (cognitive proficiency and intrinsic motivation), adapting content through a teacher agent, and evaluating quality through an automated evaluator agent. The system was evaluated through automated agent-based assessment and exploratory feedback from K-12 in-service teachers on authentic curriculum content.

EducationQ is a multi-agent dialogue framework that evaluates LLMs' teaching capabilities through simulated teacher-student interactions, testing 14 models across 1,498 questions spanning 13 disciplines and 10 difficulty levels. The framework incorporates formative assessment principles to measure pedagogical effectiveness including questioning strategies, adaptive feedback, and scaffolding behaviors.

This paper evaluates three multi-agent LLM systems that embed the Knowledge-Learning-Instruction (KLI) framework to generate secondary Math and Science learning activities, comparing them against a baseline single-agent system through teacher evaluations and LLM-as-a-judge assessments using Quality Matters K-12 standards. The study demonstrates that collaborative multi-agent systems can produce more pedagogically sound, creative, and classroom-ready activities by encoding learning science principles directly into the AI architecture.

This paper provides a comprehensive review of evaluation practices for LLM-powered Intelligent Tutoring Systems (ITSs), critically analyzing existing benchmarks and proposing three pedagogy-driven research directions for establishing unified, scalable evaluation methodologies grounded in learning science principles. It emphasizes cognitive offloading concerns, citing empirical studies showing students' over-reliance on AI tutors leading to reduced independent problem-solving skills.

This paper presents Inquizzitor, an LLM-based formative assessment agent for middle school STEM+C education that provides adaptive scaffolding through theory-driven dialogue. The framework integrates Evidence-Centered Design with Social Cognitive Theory to deliver pedagogically sound feedback and assessment grounded in cognitive science principles.

This paper presents a co-design pilot study with 19 K-12 teachers examining the classroom implementation of an AI-powered grading platform (Colleague AI) that generates rubrics and provides automated feedback on student work. Through usage logs, surveys, and interviews, the study evaluates teacher and student perspectives on AI-generated scoring versus narrative feedback, revealing trust issues with automated grading but appreciation for rapid formative feedback.

EduBench introduces a comprehensive benchmark dataset with 18,821 data points across 9 educational scenarios and 4,000+ educational contexts (covering K-12 and higher education subjects, different difficulty levels, and multiple question types), evaluated using 12 multi-dimensional metrics covering scenario adaptation, factual/reasoning accuracy, and pedagogical application. The benchmark assesses LLM capabilities in diverse educational tasks including assignment grading, study planning, tutoring, and psychological counseling, with both human and automated evaluation.

This paper introduces PBLBench, a benchmark for evaluating multimodal large language models (MLLMs) on their ability to assess STEM Project-Based Learning (PBL) outcomes, including evaluating research reports, code, experimental data, and videos using expert-validated criteria derived through the Analytic Hierarchy Process (AHP). The benchmark tests 15 MLLMs on grading and ranking PBL projects across multiple STEM disciplines with long-context multimodal inputs.

This systematic review analyzes 30 empirical studies (2021-2024) examining how prompt engineering techniques are applied to large language models in K-12 STEM education contexts, identifying common prompting strategies, model types, and evaluation methods used across teaching and learning applications.

This survey paper systematically reviews LLM applications in education from a technology-centric perspective, covering student/teacher assistance, adaptive learning, and commercial tools, while organizing related datasets, benchmarks, and identifying deployment challenges. It provides a comprehensive taxonomy of how LLMs are being applied across multiple educational contexts including tutoring, assessment, content generation, and personalized learning.

This survey paper comprehensively reviews LLM agents for education, proposing a task-centric taxonomy that categorizes agents based on teaching assistance (classroom simulation, feedback generation, curriculum design) and student support (adaptive learning, knowledge tracing, error correction). The paper analyzes architectures, datasets, benchmarks, and deployment challenges including ethics, hallucination, and overreliance.

This paper introduces The Pedagogy Benchmark, a novel dataset of 920 multiple-choice questions from Chilean teacher training exams designed to evaluate large language models' cross-domain pedagogical knowledge (CDPK) and special education needs (SEND) knowledge. The benchmark tests 97 models on their understanding of teaching strategies, assessment methods, and pedagogical concepts, with accuracies ranging from 28% to 89%.

This paper evaluates the use of pre-trained language models to automatically assess high-inference teaching practices in K-12 math classrooms and simulated teacher training settings, comparing NLP-based measurements against traditional human expert ratings across multiple dimensions of instruction quality including practices for students with special needs.

EduPlanner is an LLM-based multi-agent system that automatically generates and iteratively optimizes instructional designs (lesson plans) for mathematics curriculum, using a Skill-Tree structure to model student knowledge levels and a five-dimensional evaluation framework (CIDDP) to assess lesson plan quality.

This systematic literature review analyzes 86 studies on Intelligent Tutoring Systems (ITS) and Robot Tutoring Systems (RTS), examining their pedagogical approaches, AI techniques, engagement strategies, and effectiveness in delivering personalized K-12 instruction. The review identifies three system categories through Latent Class Analysis and evaluates technological advancements, learning outcomes, and ethical considerations.

This paper presents IDPplanner, a GPT-based tool that assists secondary school teachers in designing interdisciplinary design thinking projects aligned with Singapore's curriculum and 21st-century competencies. A workshop with 33 in-service teachers compared AI-assisted versus manual project planning using a six-dimensional rubric, finding that AI-assisted plans scored higher on curriculum alignment, design thinking application, and coherence.

This study analyzes lesson plans from 29 pre-service elementary teachers who integrated ChatGPT into science learning activities, evaluating their plans using a TPACK-based rubric and examining their perceptions about ChatGPT's pedagogical affordances and risks including student over-dependency. The research identifies how ChatGPT was envisioned for use across science domains and teaching strategies, with teachers expressing both enthusiasm about personalized learning support and concerns about accuracy and cognitive offloading.

This paper evaluates GPT-3.5 and GPT-4 models' ability to automatically assess novice human tutors' use of social-emotional learning strategies during one-on-one middle-school math tutoring sessions, comparing four prompting strategies including RAG. The work aims to enable scalable, automated evaluation of tutor competencies to inform personalized tutor training programs.

This working paper proposes a conceptual framework for an adaptive AI-based virtual tutor system that integrates psychometric assessments to identify and support K-12 students with learning difficulties (dyslexia, dyscalculia, ADHD) through personalized instruction using LLMs, visual generation models, and techniques like Bayesian knowledge tracing. The system aims to create individualized learning profiles and deliver targeted interventions based on students' cognitive abilities, learning styles, and academic skills.

This study compares lesson plans created by human curriculum designers versus two GenAI models (fine-tuned LLaMA-2-13b and customized GPT-4) in K-12 mathematics education, evaluating educator preferences across instructional components (warm-up, main task, cool-down) and grade levels. The mixed-methods analysis examines both quantitative preferences and qualitative feedback to identify strengths and limitations of AI-generated lesson plans compared to human-authored ones.

This paper presents a systematic analysis of AI-driven database systems implemented across 47 educational institutions (K-12 and higher education) over three years, evaluating their impact on personalized adaptive learning, student engagement, and academic performance. The study examines technical integration patterns, learning outcomes, and implementation challenges of AI-enhanced learning management systems.

This paper provides a comprehensive overview of AI-based adaptive learning systems, including a SWOT analysis and Force Field Analysis with 112 participants to evaluate strengths, weaknesses, opportunities, threats, and driving/resisting forces for implementing adaptive learning in education. It discusses various AI tutoring systems (AutoTutor, Affective AutoTutor) and personalized learning approaches that adjust content to individual student needs.

This paper develops a theoretical framework based on Constructive Alignment and Bloom's taxonomy to guide how educational assessment should adapt when students use AI tools like LLMs, examining whether current assessment methods remain valid and how AI affects evaluation of learning objectives at different cognitive levels.

This exploratory self-study examines ChatGPT's capabilities for answering science education questions and generating pedagogical resources like lesson plans, rubrics, and quizzes for K-12 science teachers. The study evaluates ChatGPT's content knowledge, pedagogical outputs, and potential applications in science education while highlighting concerns about epistemic authority and the need for critical evaluation of AI-generated content.

This study evaluates the RADEC (Read-Answer-Discuss-Explain-Create) instructional model integrated with AI tools to enhance science literacy among elementary school pre-service teachers in Indonesia, using student questionnaires to measure understanding of scientific concepts, motivation, problem-solving abilities, and critical thinking skills. The research focuses on preparing future teachers to use AI-enhanced pedagogy rather than directly evaluating AI systems with K-12 students.

This qualitative study investigates programming instructors' perceptions of ChatGPT's potential benefits and drawbacks for computer programming education at a university, exploring how it can support personalized learning, code debugging, and instructional efficiency while raising concerns about over-reliance, accuracy, and ethical issues.

This Indonesian case study examines English language teachers' and learners' perceptions of using AI for literary writing instruction, finding that AI helps with lesson planning, content generation, and writing skill development, but also creates dependency that limits creativity and critical thinking. The research uses questionnaire data to assess both pedagogical benefits and concerns about cognitive offloading.

This paper proposes DKVMN-KAPS, a deep learning model for knowledge tracing that predicts students' future question-answering performance by modeling individual differences in knowledge-absorption ability and problem-solving ability using hierarchical CNNs and autoencoders. The model aims to improve personalized learning by more accurately tracking students' knowledge states and providing feedback on weak knowledge areas.

This paper presents an automated LLM-based pipeline that generates knowledge components (KCs) for programming problems and uses them for knowledge tracing to predict student performance and estimate mastery levels on fine-grained skills. The framework is evaluated on real-world student code submission datasets and compared against human-written KCs and existing knowledge tracing methods.

This paper introduces CulturAIEd, an LLM-powered tool that helps K-12 teachers adapt AI literacy curricula to students' cultural contexts through Culturally Relevant Pedagogy (CRP) principles. A pilot study with four K-12 teachers explored how the tool influences their confidence and ability to design culturally responsive AI literacy activities.

This paper introduces a GenAI-powered digital assistant that uses an 'interactive mega-prompt' system to help educators create customized lesson plans tailored to student demographics, learning objectives, and teaching styles. The tool was evaluated for effectiveness through quantitative metrics (time savings) and qualitative feedback (user satisfaction) from educators across various subjects and educational levels.

This paper conducts a bibliometric analysis of 2,398 AIED research articles from 2020-2024, using keyword co-occurrence networks to identify key research topics (intelligent tutoring systems, learning analytics, NLP) and emerging frontiers (LLMs, GenAI, multimodal learning analytics, human-AI collaboration). The study maps the field's transformation but does not present original benchmarks or direct evaluation of AI systems in K-12 settings.

This position paper discusses the potential of Artificial General Intelligence (AGI) and large language models like GPT-4 to transform K-12 education through personalized learning, intelligent tutoring, adaptive assessment, and pedagogical support. It explores AGI's capabilities in understanding emotions, providing feedback, and adapting to individual student needs while addressing ethical concerns around bias, fairness, and privacy.

Edu-Values is a Chinese education values evaluation benchmark with 1,418 questions testing LLMs on seven core educational values including professional philosophy, teachers' professional ethics, education laws and regulations, cultural literacy, educational knowledge and skills, basic competencies, and subject knowledge. The benchmark evaluates 21 state-of-the-art LLMs using multiple question types including multiple-choice, multi-modal, subjective analysis, adversarial prompts, and Chinese traditional culture questions.

This study proposes a conceptual framework for using ChatGPT to enhance creative teaching skills among secondary mathematics teachers in Saudi Arabia, evaluating teachers' current proficiency levels through questionnaires and assessing the framework's appropriateness from teachers' perspectives. The research focuses on how AI can support teacher professional development rather than directly evaluating AI's impact on student learning outcomes.

This paper describes a teacher training intervention that introduced ChatGPT to primary school teachers in Indonesia, measuring knowledge gains through pre/post-tests showing a 22% average improvement. The study evaluates teacher digital literacy and familiarity with AI tools but does not directly measure impact on student learning outcomes or cognitive engagement.

This qualitative study conducted ideation workshops with 17 secondary school physical education teachers to explore their perspectives on potential AI applications and challenges in K-12 PE classes, identifying roles like operational assistant, personal trainer, group coach, and evaluator. The paper focuses on teacher attitudes and envisioned use cases rather than evaluating actual AI systems or their impact on student learning.

This is a conceptual paper that discusses the theoretical rationale, principles, and instructional implications of integrating generative AI (especially LLMs like ChatGPT) into K-12 education, focusing on pedagogical approaches and concerns about cognitive offloading. The paper does not present original empirical research, benchmarks, or evaluation methods, but rather synthesizes theoretical perspectives on how AI should be used to support rather than replace genuine learning.

This paper describes the development of a teacher guidebook for optimizing AI use in K-12 learning environments using the ADDIE instructional design model, tested with 30 teachers across 5 schools in Indonesia. The guidebook aims to provide theoretical understanding and practical applications of AI across various subjects, with 90% of teachers reporting improved understanding of AI implementation.

This paper proposes a conceptual framework for integrating AI convergence education (Intelligent Tutoring Systems, Dialogue-Based Tutoring Systems, and Exploratory Learning Environments) into K-12 technology education's problem-solving learning process. The framework structures technological problem-solving into stages (problem solving, design, development, evaluation) and maps different AI educational approaches to different knowledge types and learning stages.

This paper presents an open-source robotics learning module designed for high school and college students to build AI-powered robot companions, emphasizing hands-on learning about AI, machine learning, robotics, human-AI interaction, and AI ethics through an affordable platform (under $300).

This is a scoping review of 46 studies examining pedagogical strategies, learning tools, and assessment methods used to teach AI literacy (not evaluate AI systems) in K-12 contexts, including tools like Scratch, Python, Google's Teachable Machine, and Machine Learning for Kids. The paper focuses on how to teach students about AI concepts and computational thinking, rather than evaluating AI systems used for learning other subjects.

This is a literature review examining the broad implications of AI for teaching and learning, discussing opportunities (intelligent content, tutoring systems, personalized learning, automated grading) and challenges (infrastructure, equity, privacy, teacher readiness) across educational contexts. The paper synthesizes existing research on AI in education but does not present original benchmarks, evaluation methodologies, or empirical studies measuring AI's impact on K-12 student learning outcomes.

This paper discusses educational disparities in China between urban and rural children (migrant and left-behind children) and proposes AI tutors like ChatGPT-based Khanmigo as potential solutions to bridge the educational gap, though it does not present original evaluation research or benchmarks.

This community service paper presents a SWOT analysis of ChatGPT for English language teaching at a middle school (MTS Muhammadiyah 15 Medan), providing training to teachers on using the tool and discussing its strengths, weaknesses, opportunities, and threats in education.

This is a literature review paper that surveys AI applications in education across three main areas: student performance prediction using machine learning/deep learning, student data analysis (facial recognition and text analysis), and AI-assisted teaching cases. The paper does not present a benchmark or evaluation suite but rather synthesizes existing research on AI technologies applied to educational contexts.

This mixed-methods study surveys 100 language teachers about their perceptions of AI effectiveness in language teaching, examining what AI tools they use, challenges they face, and strategies for integration. The study focuses on teacher perspectives and practices rather than direct evaluation of AI systems or measurement of student learning outcomes.

This chapter develops a theoretical framework for 'bidirectional human-AI alignment' in education, arguing that trustworthy learning environments require both embedding human values into AI systems and equipping educators and students to interpret and critique these technologies. It provides a conceptual roadmap covering values, goals, interaction norms, and impacts on teachers, students, and institutions, with recommendations for policymakers and developers.