Orca-Math: Unlocking the potential of SLMs in Grade School Math

This paper presents Orca-Math, a 7-billion parameter language model trained on 200K synthetic math problems that achieves 86.81% accuracy on GSM8K (a grade school math benchmark) without requiring ensemble methods, code execution, or external tools. The work uses an iterative learning approach where the model practices solving problems and receives feedback from a teacher model (GPT-4).

Mathematical word problem-solving has long been recognized as a complex task for small language models (SLMs). A recent study hypothesized that the smallest model size, needed to achieve over 80% accuracy on the GSM8K benchmark, is 34 billion parameters. To reach this level of performance with smaller models, researcher often train SLMs to generate Python code or use tools to help avoid calculation errors. Additionally, they employ ensembling, where outputs of up to 100 model runs are combined t

Source

Framework Categories

Tool Types

Tags