Scoring and grading

This paper evaluates GPT-4's ability to automatically grade short answer questions in K-12 education (ages 5-16) across Science and History using a novel dataset from the Carousel quizzing platform, finding performance (Kappa 0.70) close to human-level (0.75). The study demonstrates that LLMs can reliably perform formative assessment grading tasks across multiple subjects and grade levels with minimal prompt engineering.

EduEval is a comprehensive hierarchical benchmark for evaluating LLMs in Chinese K-12 education, comprising 24 task types with over 11,000 questions organized across six cognitive dimensions (Memorization, Understanding, Application, Reasoning, Creativity, and Ethics) based on Bloom's Taxonomy and Webb's Depth of Knowledge. The benchmark uses authentic educational materials including real exam questions, classroom dialogues, student essays, and expert-designed prompts spanning primary through high school levels.

FoundationalASSIST introduces a 1.7-million interaction K-12 educational dataset with full question text, student responses, and Common Core alignment, specifically designed to evaluate whether LLMs can perform knowledge tracing (predicting student performance) and pedagogical grounding (understanding assessment item properties). The paper evaluates four frontier LLMs on these tasks, revealing significant gaps in their ability to predict student performance and understand item discrimination.

This paper introduces the AMMORE dataset of 53,000 middle-school math open-response answers from students in Africa using the Rori WhatsApp AI tutor, and evaluates LLM-based grading approaches (including chain-of-thought prompting) to improve automated scoring accuracy from 98.7% to 99.9%, demonstrating consequential validity through impacts on Bayesian Knowledge Tracing estimates of student mastery.

This paper evaluates LLM-based and statistical NLP methods for automated scoring of nationwide high school graduation essay exams in Estonia, comparing machine-generated scores against human raters across multiple rubric dimensions including content, argumentation, and language quality. The study demonstrates that automated scoring achieves reliability comparable to human raters while also examining bias, prompt injection risks, and providing personalized feedback capabilities.

This paper develops a chain-of-thought prompting approach using GPT-4 to automatically score and generate explanations for middle school Earth Science formative assessment responses, employing human-in-the-loop few-shot and active learning methods. The system evaluates open-ended short-answer responses and provides meaningful feedback to support student learning.

This paper evaluates ChatGPT's ability to solve decimal math problems, assess correctness of student self-explanations, and generate feedback within the Decimal Point learning game for middle school students, using over 5,000 real student responses. The study assesses ChatGPT's content knowledge in decimals, automated grading accuracy (75%), and pedagogical quality of generated feedback using a structured rubric.

This paper presents a co-design pilot study with 19 K-12 teachers examining the classroom implementation of an AI-powered grading platform (Colleague AI) that generates rubrics and provides automated feedback on student work. Through usage logs, surveys, and interviews, the study evaluates teacher and student perspectives on AI-generated scoring versus narrative feedback, revealing trust issues with automated grading but appreciation for rapid formative feedback.

This paper evaluates state-of-the-art VLMs and LLMs on grading over 14,000 handwritten answers from Indonesian Grade 4 students in Mathematics and English, and generating personalized feedback. The study introduces a multimodal pipeline integrating vision models for handwriting recognition and language models for rubric-based assessment in real-world, underrepresented classroom settings.

This paper compares three models (fine-tuned Mistral/GOAT, SBERT-Canberra, and GPT-4) for automatically scoring and providing qualitative feedback on middle-school students' open-ended math responses, evaluating both scoring accuracy and feedback quality using teacher judgments.

EduBench introduces a comprehensive benchmark dataset with 18,821 data points across 9 educational scenarios and 4,000+ educational contexts (covering K-12 and higher education subjects, different difficulty levels, and multiple question types), evaluated using 12 multi-dimensional metrics covering scenario adaptation, factual/reasoning accuracy, and pedagogical application. The benchmark assesses LLM capabilities in diverse educational tasks including assignment grading, study planning, tutoring, and psychological counseling, with both human and automated evaluation.

This paper introduces PBLBench, a benchmark for evaluating multimodal large language models (MLLMs) on their ability to assess STEM Project-Based Learning (PBL) outcomes, including evaluating research reports, code, experimental data, and videos using expert-validated criteria derived through the Analytic Hierarchy Process (AHP). The benchmark tests 15 MLLMs on grading and ranking PBL projects across multiple STEM disciplines with long-context multimodal inputs.

This paper investigates how assessment construct characteristics (complexity, diversity, structure) affect machine learning model performance when automatically scoring middle school students' written scientific argumentation responses across 17 assessment tasks aligned to K-12 science education standards (NGSS).

This paper develops and evaluates automated methods using large language models (GPT-4, GPT-4-mini, and ModernBERT) to assess critical thinking subskills in student-written argumentative essays, comparing zero-shot prompting, few-shot prompting, and supervised fine-tuning approaches against human coding. The work focuses on measuring higher-order reasoning skills including understanding/analyzing information, evaluating evidence, making inferences, and articulating arguments.

This paper presents a multi-agent LLM system for automatically generating multiple-choice questions (MCQs) to assess AI literacy in K-12 students (grades 7-9), using critique agents to ensure questions align with learning objectives, grade levels, and Bloom's Taxonomy. Three K-12 AI literacy teaching experts evaluated 40 generated questions using a pedagogical quality rubric.

This survey paper systematically reviews LLM applications in education from a technology-centric perspective, covering student/teacher assistance, adaptive learning, and commercial tools, while organizing related datasets, benchmarks, and identifying deployment challenges. It provides a comprehensive taxonomy of how LLMs are being applied across multiple educational contexts including tutoring, assessment, content generation, and personalized learning.

This survey paper comprehensively reviews LLM agents for education, proposing a task-centric taxonomy that categorizes agents based on teaching assistance (classroom simulation, feedback generation, curriculum design) and student support (adaptive learning, knowledge tracing, error correction). The paper analyzes architectures, datasets, benchmarks, and deployment challenges including ethics, hallucination, and overreliance.

This paper evaluates Large Language Models (ChatGPT and Llama) for automated essay scoring by comparing their grades to human raters using the ASAP dataset, finding that LLMs assign lower scores that correlate poorly with human evaluations, though they reliably detect spelling and grammar mistakes.

Grade Guard is an LLM-based framework for automated short answer grading that introduces an Indecisiveness Score to reflect grading uncertainty and enables human re-evaluation when confidence is low. The system uses fine-tuned temperature parameters and confidence-aware loss to improve grading accuracy compared to baseline LLM approaches.

This paper examines how large language models can be integrated into educational grading practices, analyzing their potential to automate assessment, provide personalized feedback, and improve consistency while addressing challenges like bias, data privacy, and ethical concerns in automated grading systems.

This paper is a systematic review of large language models (LLMs) used as virtual tutors in education, examining applications including automatic question generation, student assessment, feedback provision, and personalized learning across K-12 and higher education contexts. The review analyzes how LLMs like GPT-3 and BERT are deployed for educational tasks such as generating explanations, grading answers, and providing tutoring support.

This white paper examines ethical implications of AI-powered tools in educational measurement, focusing on automated scoring, personalized feedback, and data analytics while addressing concerns about validity, reliability, transparency, fairness, algorithmic bias, and equity in educational assessment contexts.

This paper proposes using DeBERTa with Adversarial Weights Perturbation and Metric-specific AttentionPooling to improve automated essay scoring specifically for English Language Learners (ELLs), focusing on optimizing hyperparameters to better evaluate writing proficiency.

This paper evaluates the use of pre-trained language models to automatically assess high-inference teaching practices in K-12 math classrooms and simulated teacher training settings, comparing NLP-based measurements against traditional human expert ratings across multiple dimensions of instruction quality including practices for students with special needs.

This paper develops an ensemble NLP model combining BERT, RoBERTa, BART, DeBERTa, and T5 to automatically assess English Language Learners' (grades 8-12) writing across six linguistic dimensions: cohesion, syntax, vocabulary, phraseology, grammar, and conventions. The system uses stacking techniques with LightGBM and Ridge regression to provide automated scoring and feedback on student essays.

This paper introduces 'Marking', an automated grading system that goes beyond binary scoring by highlighting correct/incorrect/irrelevant segments in student responses and identifying omissions from gold answers. The authors use NLI-based transformer models (BERT, RoBERTa) and create the BioMarking dataset with biology subject matter expert annotations to train and evaluate this fine-grained feedback system.

This paper investigates using AI technologies (NLP, chatbots, adaptive learning systems) to automate thematic control and criteria-based assessment of foreign language writing skills, evaluating student texts for grammar, vocabulary, coherence, relevance, structure, and stylistic correctness in real-time. The study examines AI's applicability for conducting various types of educational assessment (preliminary, current, intermediate, thematic, final) with focus on automated grading and feedback provision.

This paper proposes an automated essay scoring system using sentence-BERT embeddings and Bi-LSTM networks to evaluate student short-answer responses, focusing on capturing coherence and cohesion in essays. The system is tested on the ASAP Kaggle dataset and a domain-specific dataset with 2,500 student responses, achieving an average QWK score of 0.76.

This paper compares machine learning algorithms (including random forest, which achieved 97.67% accuracy) for automated essay scoring/grading and deploys the best model as a web application using Flask. The work addresses automated assessment of written responses to reduce teacher burden in grading essays.

This paper proposes using Natural Language Processing word embeddings to automatically score Cloze test responses by measuring semantic similarity between student answers and correct answers, validated with data from Brazilian K-12 students. The approach aims to make large-scale reading comprehension assessment more efficient by moving beyond exact-match scoring to accept semantically similar responses.

This paper explores the feasibility of using Natural Language Processing, Machine Learning, and AI to optimize assessment and feedback for junior high school English writing, aiming to improve efficiency and provide personalized feedback to students. The study implements technology-based writing assessment tools and evaluates their effectiveness in enhancing writing instruction quality and student motivation.

This paper proposes an automated Descriptive Answer Evaluation System (DAES) using NLP and machine learning to evaluate student-written descriptive answers, aiming to reduce manual grading workload and provide faster feedback. The system focuses on semantic similarity, word order, sentence sequence, and spell-checking for assessment.

This paper presents a machine learning and NLP-based system for automatically evaluating subjective (essay) answers in educational settings, aiming to provide automated scoring and feedback that aligns with human evaluators while reducing teacher workload. The system uses text preprocessing, feature extraction, and ML algorithms trained on annotated datasets to assess open-ended student responses.

This paper systematically benchmarks ChatGPT (GPT-3.5) and GPT-4 against human tutors across six programming education scenarios (program repair, hint generation, grading feedback, pair programming, contextualized explanation, and task synthesis) using introductory Python problems and real-world buggy student code. The evaluation uses expert annotations to assess pedagogical quality across scenarios relevant to digital tutoring, teaching assistance, and peer learning.

This paper presents AERA, a framework that uses ChatGPT to generate rationales for automated scoring of student short-answer responses, then distills this capability into a smaller model that can simultaneously score answers and provide explanations. The approach is evaluated on a benchmark dataset of student responses to science questions, showing improved scoring accuracy and comparable rationale quality to ChatGPT.

This paper develops methods to detect boundaries between human-written and AI-generated content in hybrid essays where students may have used LLMs like ChatGPT to complete portions of their writing assignments. The approach identifies transition points in student essays to help educators detect AI assistance in academic writing.

This paper evaluates the docimological quality of multiple-choice questions (MCQs) automatically generated by Large Language Models (GPT-based) in computer science and medicine domains, comparing them against established MCQ item-writing guidelines. The study analyzes LLM-generated questions for pedagogical quality including clarity, plausibility of distractors, and alignment with educational best practices.

This paper presents GradeHITL, an LLM-based automated grading framework with human-in-the-loop that enables AI to ask clarifying questions about rubrics to human experts, dynamically refining grading standards for short-answer assessment. The system is evaluated on mathematics teaching knowledge questions, using reinforcement learning to filter high-quality clarification questions.

This paper explores using large language models (GPT-4 and Llama 2) to automatically generate and evaluate German reading comprehension multiple-choice test items, proposing a new evaluation metric called 'text informativity' based on answerability and guessability. The work evaluates both human and LLM-based assessment of generated items using zero-shot prompting.

This paper evaluates Large Language Models (LLMs) for automatically assessing the correctness of students' line-by-line natural language explanations of computer code (Java programs) in introductory CS courses, comparing LLM performance against fine-tuned encoder-based semantic similarity models. The work focuses on automated short answer grading in the programming domain to support self-explanation activities that improve code comprehension.

This paper compares AI-generated assessment scores with Teaching Assistant scores for evaluating design thinking posters created by 13-14 year-old secondary students in Singapore, examining agreement across dimensions like empathy, pain points, and visual communication. The study investigates teacher preferences for AI versus human grading and explores hybrid assessment models for scalable evaluation in design thinking education.

GradeOpt is an LLM-powered automatic short-answer grading framework that uses multi-agent systems to grade open-text responses and automatically optimize grading guidelines through self-reflection. The system is evaluated on datasets measuring teachers' pedagogical knowledge and students' learning progress in mathematics and physical science.

This position paper critiques the overreliance on inter-rater reliability (IRR) metrics like Cohen's kappa for validating human annotations in educational AI systems, particularly for automated assessment tasks like grading open responses and classifying tutor dialogue moves. It proposes alternative evaluation methods (multi-label schemes, expert-based approaches, close-the-loop validity) that better capture educational validity and learning impact.

This paper evaluates whether AI-generated multiple-choice questions (MCQs) can be accurately aligned with Bloom's Taxonomy cognitive levels using various classification models, including DistilBERT, to improve automated question generation in the OneClickQuiz Moodle plugin. The study develops a dataset of 3,691 questions categorized by Bloom's levels and assesses model effectiveness in classifying questions by cognitive complexity.

This paper explores prompting strategies for large language models to jointly perform automated essay scoring and generate individualized feedback for student essays, evaluating both the scoring accuracy and helpfulness of generated feedback. The work uses the ASAP dataset of student essays and compares zero-shot and few-shot prompting approaches for providing explanatory feedback to help students improve their writing.

This paper comprehensively evaluates nine Automated Essay Scoring (AES) methods across 25,000+ student essays, examining the interplay between predictive accuracy, fairness (bias across demographic groups), and generalizability in both prompt-specific and cross-prompt settings. The study reveals that prompt-specific models achieve higher accuracy but exhibit greater bias toward students of different economic statuses compared to cross-prompt models.

This paper evaluates the effectiveness of multiple large language models (ChatGPT, Llama, Aya, Jais, ACEGPT) for automated essay scoring in Arabic using the AR-AES dataset, comparing zero-shot, few-shot, and fine-tuning approaches with various prompt engineering strategies. The study finds that while ACEGPT performs best among LLMs (QWK=0.67), a smaller BERT-based model outperforms all tested LLMs (QWK=0.88).

This systematic literature review examines automated essay scoring (AES) systems from 2018-2023, analyzing their accuracy, scalability, robustness, and functionality in real-world educational contexts, particularly for English writing assessment. The review identifies strengths and weaknesses of current AES systems and their limitations in meeting teachers' and students' needs in authentic teaching scenarios.

This paper proposes LCES, a zero-shot automated essay scoring method that uses large language models to perform pairwise comparisons between essays and aggregates these comparisons into continuous scores using RankNet, avoiding the need for prompt-specific training data.

This paper develops and evaluates machine learning models for automatically scoring formal register (grammar, vocabulary, language proficiency) in narrative essays written by K-12 students in Portuguese, comparing automated scores against human annotators. The study creates a dataset of 327 annotated student narrative essays and tests various ML algorithms to predict human ratings.