New research conducted by a professor at University of Pennsylvania’s Wharton School found that the artificial intelligence-driven chatbot GPT-3 was able to pass the final exam for the school’s Master of Business Administration (MBA) program.
Professor Christian Terwiesch, who authored the research paper “Would Chat GPT3 Get a Wharton MBA? A Prediction Based on Its Performance in the Operations Management Course,” said that the bot scored between a B- and B on the exam.
The bot’s score, Terwiesch wrote, shows its “remarkable ability to automate some of the skills of highly compensated knowledge workers in general and specifically the knowledge workers in the jobs held by MBA graduates including analysts, managers, and consultants.”
The bot did an “amazing job at basic operations management and process analysis questions including those that are based on case studies,” Terwiesch wrote in the paper, which was published on Jan. 17. He also said the bot’s explanations were “excellent.”
The bot is also “remarkably good at modifying its answers in response to human hints,” he concluded.
Terwiesch’s findings come as educators become increasingly concerned that AI chatbots could inspire cheating. Although chatbots are not a new technology, ChatGPT exploded on social media in late 2022. Earlier this month, New York City’s Department of Education announced a ban on ChatGPT from its schools’ devices and networks.
Much of the debate is centered around ChatGPT’s conversational speaking style and coherent, topical response style, which makes it difficult to distinguish from human responses.
Experts who work in both artificial intelligence and education have acknowledged that bots like ChatGPT could be a detriment to education in the future. But in recent interviews, some educators and experts they weren’t concerned — yet.
Terwiesch did not immediately respond to a request for further comment. A spokesperson for artificial intelligence startup OpenAI, which created the bot, also did not immediately respond to a request for comment Monday.
The GPT-3 model used in the experiment appears to be an older sibling of the most recent ChatGPT bot that has become a controversial topic among educators and those who work in the field of AI. ChatGPT, the newest version, “is fine-tuned from a model in the GPT-3.5 series,” according to OpenAI’s website.
While Chat GPT3’s results were impressive, Terwiesch noted that Chat GPT3 “at times makes surprising mistakes in relatively simple calculations at the level of 6th grade Math.”
The present version of Chat GPT is “not capable of handling more advanced process analysis questions, even when they are based on fairly standard templates,” Terwiesch added. “This includes process flows with multiple products and problems with stochastic effects such as demand variability.”
Still, Terwiesch said ChatGPT3’s performance on the test has “important implications for business school education, including the need for exam policies, curriculum design focusing on collaboration between human and AI, opportunities to simulate real world decision making processes, the need to teach creative problem solving, improved teaching productivity, and more.”
