A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity

This paper presents a comprehensive evaluation of ChatGPT across 23 datasets covering 8 NLP tasks, testing multitask, multilingual, and multimodal capabilities, with focus on reasoning accuracy, hallucination problems, and interactive prompt engineering. The evaluation includes logical reasoning, commonsense reasoning, machine translation, summarization, and other general NLP tasks in multiple languages.

This paper proposes a framework for quantitatively evaluating interactive LLMs such as ChatGPT using publicly available data sets. We carry out an extensive technical evaluation of ChatGPT using 23 data sets covering 8 different common NLP application tasks. We evaluate the multitask, multilingual and multi-modal aspects of ChatGPT based on these data sets and a newly designed multimodal dataset. We find that ChatGPT outperforms LLMs with zero-shot learning on most tasks and even outperforms fin

Source

Framework Categories

Tool Types

Tags