MalAlgoQA: Pedagogical Evaluation of Counterfactual Reasoning in Large Language Models and Implications for AI in Education

MalAlgoQA introduces a dataset of K-12 mathematics and reading comprehension questions (grades 3-11) with rationales for both correct answers and incorrect answer choices ('malgorithms'), designed to evaluate LLMs' counterfactual reasoning abilities by testing whether they can identify the flawed reasoning that leads to wrong answers. The paper evaluates multiple state-of-the-art LLMs and finds significant performance drops when identifying malgorithms versus correct algorithms, with important implications for AI tutoring systems that need to diagnose student misconceptions.

This paper introduces MalAlgoQA, a novel dataset designed to evaluate the counterfactual reasoning capabilities of Large Language Models (LLMs) through a pedagogical approach. The dataset comprises mathematics and reading comprehension questions, each accompanied by four answer choices and their corresponding rationales. At the heart of MalAlgoQA are ``malgorithms'' - rationales behind incorrect answer choices that represent flawed yet logically coherent reasoning paths. These malgorithms serve

Source

Framework Categories

Tool Types

Tags