A new article, by Daniel Martin Katz, Michael James Bonnarito, Shang Gao and Pablo Arrendondo. job here on SSRN, report on the outstanding performance of ChatGPT-4 on the bar exam (multi-state test and multi-state performance). Congratulations, ChatGPT!
From the summary:
In this paper, we experimentally evaluate the zero-stroke performance of a pre-release version of GPT-4 compared to previous generations of GPT on the entire Uniform Bar Examination (UBE), including not only the examination of the multi-state multiple-choice bar (MBE), but also the open components Multistate Essay Exam (MEE) and Multistate Performance Test (MPT). On the MBE, GPT-4 significantly outperforms human candidates and previous models, demonstrating a 26% increase over ChatGPT and beating humans in five out of seven areas. On the MEE and MPT, which have yet to be evaluated by researchers, GPT-4 scores an average of 4.2/6.0 compared to much lower scores for ChatGPT. Ranked across UBE components, the way a human taste taker would be, the GPT-4 scores around 297 points, well above the pass mark for all UBE jurisdictions. These findings document not only the rapid and remarkable advance in the performance of major linguistic models in general, but also the potential of these models to support the delivery of legal services in society.
