Pedagogy of Generated Outputs: Evaluating Whether AI-Created Educational Content Supports Genuine Learning

A growing body of research — 99 papers in this analysis — examines whether AI-generated educational content meets the pedagogical standards required for effective teaching and learning. The central finding is sobering: while large language models (LLMs) can produce grammatically fluent, contextually relevant educational materials at impressive speed and scale, they frequently fail on the dimensions that matter most for learning. GPT-4 explanations match their intended educational level only 50% of the time, AI-generated multiple-choice question (MCQ) distractors align poorly with actual student misconceptions despite appearing plausible to educators, and LLM-based tutors exhibit what researchers call 'compulsive intervention bias' — predicting high-intervention moves 95.8% of the time when effective tutoring requires silence 41.7% of the time.

This gap between surface-level quality and genuine pedagogical effectiveness carries significant implications for education systems in low- and middle-income countries (LMICs), where AI-EdTech tools are increasingly promoted as solutions to teacher shortages and resource constraints. The research reveals a fundamental tension at the heart of current LLM design: models optimised for helpfulness and user satisfaction actively contradict the pedagogical need for productive struggle, scaffolded support, and learner independence. Perhaps most concerning is evidence from a field experiment with approximately 1,000 Turkish secondary school students showing that AI-assisted learners solved 48% more practice problems correctly but scored 17% lower on unassisted tests — a clear demonstration that improved immediate performance can mask undermined learning.

The field has made meaningful progress in developing evaluation frameworks grounded in learning science — including Bloom's Taxonomy alignment, scaffolding quality rubrics, and Item Response Theory (IRT) validation — but critical gaps remain. Almost no studies measure actual long-term learning outcomes, and the overwhelming focus on English-language, Western-curriculum contexts means we know very little about how these findings translate to the diverse linguistic and cultural settings where AI-EdTech is most needed.