Methods

Overview

This project maps AI/LLM benchmarks and evaluations to a quality framework for K-12 education, with a focus on low- and middle-income countries (LMICs). The goal is to identify what we can measure about AI tools for education — and where the gaps are.

The pipeline has four stages: discovery, classification, enrichment, and synthesis. Each stage is automated but human-reviewed.

Stage 1: Paper Discovery

We search for papers and datasets from two primary sources:

We run over 80 targeted search queries designed to cover every part of the quality framework — from broad queries like "LLM evaluation K-12 education" to specific ones like "automated essay scoring evaluation" or "zone of proximal development AI tutoring". Results are deduplicated by URL to build a comprehensive corpus.

Stage 2: Two-Stage Classification

Every paper is classified against 11 quality components, organised into 6 areas, and 3 tool types. Classification uses a two-stage pipeline:

Stage 3: Enrichment

After classification, we enrich each paper with additional metadata from Semantic Scholar's batch API:

• TLDR summaries — AI-generated one-line paper summaries from Semantic Scholar
• Citation counts — how often the paper has been cited
• PDF URLs — direct links to open-access versions where available
• Publication venues — journal or conference names
• Publication year — for time-based filtering

Stage 4: Landscape Summaries

For each quality component, tool type, and concern theme, we produce an AI-generated landscape summary synthesising findings across all high-relevance papers (scored ≥7/10):

1. Section extraction — we extract key sections (abstract, introduction, results, discussion, conclusions) from each paper's full text
2. Batch synthesis — extracted sections are sent to Claude in batches, which synthesises findings into structured analyses covering: executive summary, key findings, what's measured, what's missing, notable studies, LMIC context, and recommendations
3. Multi-batch merging — when there are too many papers for a single LLM context window, multiple batch reports are generated and then merged into a single coherent summary

The Quality Framework

Papers are mapped to 11 quality components across 6 areas. These components define what a high-quality AI tool for K-12 education should be measured on:

Concern Themes

In addition to the quality framework, we identify 5 cross-cutting concern themes — risks and challenges that span multiple framework categories. Papers are matched to concerns via keyword search over their title, summary, and full text.

Tools & Technology

Limitations

• Search coverage — while we query 80+ search terms across multiple sources, some relevant papers may be missed if they don't match our query set or aren't indexed by Semantic Scholar or HuggingFace.
• Classification accuracy — LLM-based classification is not perfect. Some papers may be mis-classified or assigned incorrect relevance scores. The heuristic + LLM two-stage approach reduces errors but doesn't eliminate them.
• Synthesis quality — landscape summaries are AI-generated and may contain inaccuracies or miss nuances present in the original papers. They should be treated as starting points for further investigation, not as definitive reviews.
• Temporal bias — the corpus reflects what's available at the time of search. Newly published papers won't appear until the pipeline is re-run.