Pedagogy of generated outputs

EduEval is a comprehensive hierarchical benchmark for evaluating LLMs in Chinese K-12 education, comprising 24 task types with over 11,000 questions organized across six cognitive dimensions (Memorization, Understanding, Application, Reasoning, Creativity, and Ethics) based on Bloom's Taxonomy and Webb's Depth of Knowledge. The benchmark uses authentic educational materials including real exam questions, classroom dialogues, student essays, and expert-designed prompts spanning primary through high school levels.

FEANEL is a benchmark for evaluating LLMs' ability to provide fine-grained error analysis and pedagogical feedback on K-12 English writing, comprising 1,000 student essays with expert-annotated errors categorized by type, severity, and explanations. The benchmark specifically assesses whether AI systems can identify writing errors and provide educationally meaningful, interpretable feedback to support student learning.

This paper presents LearnLM-Tutor, a fine-tuned Gemini model for educational use, and introduces a comprehensive evaluation framework spanning seven diverse benchmarks (quantitative, qualitative, automatic, and human evaluations) grounded in learning science principles to assess pedagogical quality in K-12 AI tutoring systems. The work includes real-world deployment at Arizona State University and systematic evaluation of pedagogical dimensions including Socratic dialogue, adaptive scaffolding, and learning-centered interactions.

EduGuardBench is a dual-component benchmark designed to evaluate LLMs acting as simulated teachers, measuring both pedagogical fidelity (role-playing accuracy, teaching competence) and adversarial safety (resistance to jailbreaking, handling of academic misconduct requests). The benchmark identifies harmful teaching behaviors (incompetence, indolence, offensiveness) and uses persona-based adversarial prompts to test ethical boundaries specific to educational contexts.

This paper introduces GuideEval, a benchmark that evaluates LLMs' ability to provide adaptive Socratic tutoring by assessing three pedagogical phases: perceiving learner states (confusion, comprehension, errors), orchestrating appropriate instructional strategies, and eliciting productive reflections. The benchmark is grounded in authentic K-12 educational dialogues and specifically measures whether LLMs can dynamically adjust their guidance based on student cognitive states rather than just generate generic responses.

TutorBench is a purpose-built benchmark with 1,490 expert-curated samples evaluating LLMs on three core tutoring skills: generating adaptive explanations, providing actionable feedback, and creating effective hints for high-school and AP-level content. The benchmark uses sample-specific rubrics and LLM-judge evaluation to assess 16 frontier models, finding none exceed 56% overall performance and all achieve less than 60% pass rate on criteria related to guiding, diagnosing, and supporting students.

FACET is a teacher-facing LLM-based multi-agent system that generates personalized mathematics worksheets for grade 8 students by modeling learner profiles (cognitive proficiency and intrinsic motivation), adapting content through a teacher agent, and evaluating quality through an automated evaluator agent. The system was evaluated through automated agent-based assessment and exploratory feedback from K-12 in-service teachers on authentic curriculum content.

Mentigo is an AI mentor agent system designed to guide middle school students through creative problem solving (CPS) processes, providing scaffolding, personalized feedback, and Socratic questioning based on real classroom mentor-student interactions. The system was evaluated through comparative experiments with 12 students and reviewed by expert teachers, demonstrating improvements in student engagement and creative outcomes.

This paper evaluates the effectiveness of LLM-generated analogies for teaching scientific concepts (biology and physics) through controlled in-class tests with high school students and classroom field studies with teachers. The study measures student understanding, learning outcomes, potential over-reliance, and teacher satisfaction with LLM-generated analogies, culminating in the development of a practical system for teachers to generate and refine teaching analogies.

This paper develops and evaluates a pedagogical framework for LLM-driven feedback generation in secondary school Python programming education, aligning automated feedback with established pedagogical principles like mastery adaptation and progress-based scaffolding. Through mixed-method evaluation with eight secondary school computer science teachers, the study assesses how well LLM-generated feedback adheres to pedagogical standards compared to human teacher feedback.

This paper develops a chain-of-thought prompting approach using GPT-4 to automatically score and generate explanations for middle school Earth Science formative assessment responses, employing human-in-the-loop few-shot and active learning methods. The system evaluates open-ended short-answer responses and provides meaningful feedback to support student learning.

EducationQ is a multi-agent dialogue framework that evaluates LLMs' teaching capabilities through simulated teacher-student interactions, testing 14 models across 1,498 questions spanning 13 disciplines and 10 difficulty levels. The framework incorporates formative assessment principles to measure pedagogical effectiveness including questioning strategies, adaptive feedback, and scaffolding behaviors.

This paper evaluates three multi-agent LLM systems that embed the Knowledge-Learning-Instruction (KLI) framework to generate secondary Math and Science learning activities, comparing them against a baseline single-agent system through teacher evaluations and LLM-as-a-judge assessments using Quality Matters K-12 standards. The study demonstrates that collaborative multi-agent systems can produce more pedagogically sound, creative, and classroom-ready activities by encoding learning science principles directly into the AI architecture.

ConvoLearn introduces a dataset of 1,250 constructivist tutor-student dialogues in middle school Earth Science, grounded in knowledge-building theory across six pedagogical dimensions (cognitive engagement, formative assessment, accountability, cultural responsiveness, metacognition, and power dynamics). The paper demonstrates that fine-tuning LLMs on this dataset shifts model behavior toward constructivist teaching strategies, with the fine-tuned Mistral-7B significantly outperforming base models and Claude Sonnet in teacher evaluations.

This paper evaluates ChatGPT's ability to solve decimal math problems, assess correctness of student self-explanations, and generate feedback within the Decimal Point learning game for middle school students, using over 5,000 real student responses. The study assesses ChatGPT's content knowledge in decimals, automated grading accuracy (75%), and pedagogical quality of generated feedback using a structured rubric.

This paper presents a framework for finetuning LLMs to generate grade-appropriate educational content across six grade levels (lower elementary through adult), evaluating the pedagogical quality and age-appropriateness of AI-generated explanations using readability metrics and human evaluation with 208 participants.

CODAE is a framework that fine-tunes open-source LLMs for AI tutoring by augmenting real student-tutor dialogues with Chain-of-Thought prompting to improve pedagogical quality. The paper addresses three key limitations (over-compliance, low response adaptivity, and threat vulnerability) and evaluates models on their ability to provide step-by-step guidance without prematurely revealing answers.

This paper provides a comprehensive review of evaluation practices for LLM-powered Intelligent Tutoring Systems (ITSs), critically analyzing existing benchmarks and proposing three pedagogy-driven research directions for establishing unified, scalable evaluation methodologies grounded in learning science principles. It emphasizes cognitive offloading concerns, citing empirical studies showing students' over-reliance on AI tutors leading to reduced independent problem-solving skills.

This paper systematically compares AI-simulated tutoring dialogues with authentic human teacher-student dialogues using IRF coding and Epistemic Network Analysis, finding that human dialogues show superior pedagogical questioning, feedback, and cognitively-guided interaction patterns, while AI dialogues exhibit structural simplification and behavioral convergence. The work directly evaluates the pedagogical quality and instructional effectiveness of LLM-generated one-on-one tutoring interactions.

MetaCLASS introduces a framework and benchmark for evaluating LLM-based metacognitive tutoring that explicitly targets self-regulated learning processes (planning, monitoring, debugging, evaluation) through 11 interpretable coach moves, including productive restraint. The paper generates 1,015 annotated tutoring conversations and benchmarks nine LLMs on predicting appropriate metacognitive coaching actions, revealing systematic compulsive intervention bias where models fail to recognize when silence is pedagogically optimal.

This paper develops and evaluates a reinforcement learning framework for automatically generating pedagogically valid feedback for incorrect student answers in math education, using GPT-4 to evaluate feedback quality according to a rubric measuring both correctness and alignment with educational goals. The work demonstrates that fine-tuning Llama 2 with direct preference optimization significantly improves feedback quality across correctness and pedagogical alignment dimensions.

This paper benchmarks whether large language models (LLMs) can replicate the adaptivity of intelligent tutoring systems by systematically removing context components from 75 real-world tutoring scenarios and evaluating how three LLMs respond to student errors, knowledge states, and pedagogical requirements. The study finds that even the best-performing LLM only marginally mimics ITS adaptivity, with concerning patterns like GPT-4o providing overly direct feedback instead of effective Socratic questioning.

This paper explores using GPT-4 to automatically contextualize math problems in CTAT (an intelligent tutoring system) to align with individual student interests at scale, aiming to increase engagement and learning outcomes. The authors use iterative prompt engineering to personalize problem contexts while preserving difficulty and pedagogical intent.

This study examines how conversational AI (LLM-based) supports K-12 mathematics educators during lesson preparation by analyzing educator-AI interactions across different educational contexts. The research evaluates AI's ability to adapt responses to educators' instructional needs, assessing accuracy, relevance, and usefulness of AI-generated pedagogical content and guidance.

This paper introduces the task of generating reasoning trajectories for Socratic debugging conversations where AI tutors guide novice programmers to identify and correct their own programming misconceptions through cognitive dissonance rather than direct correction. The work includes a manually annotated dataset of debugging problems with reasoning trajectories and evaluates LLM-generated Socratic conversations.

This is a three-day lesson plan for 9th-12th grade English Language Arts that uses AI writing assistants to help students analyze their writing through AI-enhanced rubrics, receive automated feedback, and engage in AI-informed peer review to revise their work. The lesson directly evaluates how AI tools impact student writing revision processes and self-assessment skills in an actual classroom setting.

This paper presents a randomized controlled trial (n=23) evaluating GPT-4's ability to personalize science texts for middle school students by profiling their learning preferences and rewriting content accordingly. The study measures student preference for personalized versus non-personalized texts, finding significant preference for aligned content.

EduAlign proposes a reinforcement learning framework to align large language models with pedagogical principles of helpfulness, personalization, and creativity for AI tutoring. The framework includes HPC-RM, a multi-dimensional reward model trained on 8k educational interactions, and uses GRPO to fine-tune LLMs to become more pedagogically effective AI tutors.

This paper develops and evaluates a generative AI-based system using ChatGPT to automatically assess and provide feedback on student argumentation writing by extracting argument components (claims, data, warrants, etc.). The system compares AI performance against human expert evaluation across scientific and non-scientific contexts for K-12 argumentation training.

This paper investigates whether LLMs' generation probabilities and error patterns align with student misconception patterns in multiple-choice questions, using real-world student response data across 3,202 MCQs. The study finds moderate correlation between LLM probabilities and student distractor selections, with LLMs frequently choosing the same incorrect answers that commonly mislead students.

This paper develops and evaluates a ChatGPT-based system that automatically generates personalized reading comprehension exercises (passages and multiple-choice questions) for middle school English learners in China, with extensive human evaluation by students, teachers, and native speakers demonstrating the quality meets or exceeds human-written materials.

This paper evaluates the quality and usefulness of educational questions generated by large language models (specifically GPT-3/InstructGPT) using controllable text generation with Bloom's taxonomy and difficulty levels, validated through teacher assessments across two domains. The study demonstrates that LLM-generated questions are judged by teachers as high quality and sufficiently useful for classroom use.

This paper develops and evaluates scaffolding techniques for multi-modal intelligent tutoring systems that guide children in describing images for language learning, using GPT-4V with pedagogical instructions grounded in four learning theories. The work constructs a seven-dimension rubric to evaluate the scaffolding process both manually and automatically.

This paper evaluates the multilingual performance of large language models (GPT-4o, Gemini, Claude, Llama, Mistral) on four K-12 educational tasks: identifying student misconceptions, providing targeted feedback, interactive tutoring, and grading translations across nine languages including English, Mandarin, Hindi, Arabic, German, Farsi, Telugu, Ukrainian, and Czech.

EduBench introduces a comprehensive benchmark dataset with 18,821 data points across 9 educational scenarios and 4,000+ educational contexts (covering K-12 and higher education subjects, different difficulty levels, and multiple question types), evaluated using 12 multi-dimensional metrics covering scenario adaptation, factual/reasoning accuracy, and pedagogical application. The benchmark assesses LLM capabilities in diverse educational tasks including assignment grading, study planning, tutoring, and psychological counseling, with both human and automated evaluation.

This paper introduces PBLBench, a benchmark for evaluating multimodal large language models (MLLMs) on their ability to assess STEM Project-Based Learning (PBL) outcomes, including evaluating research reports, code, experimental data, and videos using expert-validated criteria derived through the Analytic Hierarchy Process (AHP). The benchmark tests 15 MLLMs on grading and ranking PBL projects across multiple STEM disciplines with long-context multimodal inputs.

TeachLM is a fine-tuned LLM for one-on-one tutoring, trained on 100,000 hours of authentic student-tutor interactions from Polygence. The paper introduces a novel multi-turn evaluation protocol using synthetic dialogues and demonstrates improvements in pedagogical interactions including doubled student talk time, better questioning strategies, and increased dialogue turns.

This paper evaluates LLMs (GPT-4o, Llama-3-8B) as teachers generating hints for simulated LLM students solving high-school mathematics problems, measuring hint quality and students' ability to self-correct errors. The study compares different prompting strategies and model configurations to assess pedagogical effectiveness of AI-generated hints.

This paper develops and evaluates an LLM-powered pipeline (using GPT-4o) that transcreates English reading comprehension passages and questions to align with individual EFL students' interests, maintaining linguistic complexity while adapting semantic content. A controlled experiment with South Korean EFL learners demonstrates that personalized, interest-aligned reading materials improve both comprehension and motivation retention compared to non-personalized materials.

This paper evaluates LLMs as tutors for English as a Foreign Language (EFL) writing education, proposing pedagogically-grounded metrics to assess the quality and characteristics of AI-generated essay feedback through expert evaluation and student learning outcome measurements. The work includes developing score-based prompting methods to improve feedback effectiveness for EFL learners.

This paper presents a system that uses LLMs to transform multimodal data (eye-tracking, assessment scores, and teaching standards) into actionable reading assessment reports for K-12 teachers. The system employs unsupervised clustering to identify reading behavior patterns and LLMs to generate teacher-friendly reports that are evaluated by both LLM experts and human educators.

This systematic review analyzes 30 empirical studies (2021-2024) examining how prompt engineering techniques are applied to large language models in K-12 STEM education contexts, identifying common prompting strategies, model types, and evaluation methods used across teaching and learning applications.

This paper presents a multi-agent LLM system for automatically generating multiple-choice questions (MCQs) to assess AI literacy in K-12 students (grades 7-9), using critique agents to ensure questions align with learning objectives, grade levels, and Bloom's Taxonomy. Three K-12 AI literacy teaching experts evaluated 40 generated questions using a pedagogical quality rubric.

This survey paper systematically reviews LLM applications in education from a technology-centric perspective, covering student/teacher assistance, adaptive learning, and commercial tools, while organizing related datasets, benchmarks, and identifying deployment challenges. It provides a comprehensive taxonomy of how LLMs are being applied across multiple educational contexts including tutoring, assessment, content generation, and personalized learning.

This survey paper comprehensively reviews LLM agents for education, proposing a task-centric taxonomy that categorizes agents based on teaching assistance (classroom simulation, feedback generation, curriculum design) and student support (adaptive learning, knowledge tracing, error correction). The paper analyzes architectures, datasets, benchmarks, and deployment challenges including ethics, hallucination, and overreliance.

This paper systematically benchmarks ChatGPT (GPT-3.5) and GPT-4 against human tutors across six programming education scenarios (program repair, hint generation, grading feedback, pair programming, contextualized explanation, and task synthesis) using introductory Python problems and real-world buggy student code. The evaluation uses expert annotations to assess pedagogical quality across scenarios relevant to digital tutoring, teaching assistance, and peer learning.

This paper advocates for theoretically-grounded approaches to designing LLM-generated feedback in Intelligent Tutoring Systems by reviewing decades of ITS research on feedback generation methods (expert-created models, data-driven models, and LLMs) and proposing how learning science principles should guide modern LLM-based feedback design and evaluation.

This paper presents PyFiXV, a system that uses large language models (Codex) to automatically generate feedback for Python syntax errors in introductory programming courses, including both fixed code and natural language explanations. The system incorporates a validation mechanism to ensure high-precision feedback before sharing with students.

This paper develops and evaluates the StAP-tutor, an LLM-based system that generates next-step hints for introductory Python programming exercises, comparing different prompt engineering approaches and assessing hint quality through student experiments and expert evaluation. The work focuses on creating pedagogically effective automated feedback that provides specific guidance without revealing complete solutions.

This paper evaluates five large language models' ability to generate educational questions at different Bloom's taxonomy cognitive levels using various prompting strategies, with both expert human evaluation and LLM-based automated evaluation of question quality, linguistic correctness, and pedagogical relevance.