Nowadays, many different GenAI applications can assist students in writing tasks. Assisting can be just brainstorming about ideas or helping structuring an essay, but it can also be writing the full length essay with minimum to no student intervention. The latter is probably not the smartest choice as probably every school institution has GenAI policies, usually prohibiting to present the AI’s work as your own product. That’s why I was surprised while reading the course manual of Digital Business, (part of the RSM’s BA program) last academic year. I needed to write two essays about digital topics, of which the first essay had to be co-written with a GenAI tool. I chose my familiar friend, ChatGPT, to assist me in taking on this challenge. Indeed, it became more of a challenge than I initially expected it to be.
A promising start
Every topic that the essay had to include was summarized in the assignment description. The mandatory first step was to copy/paste the description and use it as the first prompt for the GenAI application. At first glance, ChatGPT answered the prompt with already a quite impressive essay. But some things were not sufficient according to the essay description and my own preferences. The essay was way too long, did not include sources and it had more a kind of question-answer format. After this, I asked ChatGPT for more theory and explanations in order to develop my own desired theories that I wanted to include in the answer. This learning process was really nice as the interaction with the AI was on a deeper level: understanding core concepts.
Shift in focus
Eventually, I prompted my own desired answer with bullet points and let my companion write the full length essay. Here is where my focus shifted from understanding theory to more superficial tasks such as finding sources, citing, essay structure and much more. Finding sources and citations has caused me the highest levels of frustration, due to my wrong expectations of ChatGPT’s capabilities.
Problems, problems and more problems.
Initially, I thought that finding references with Chat was a piece of cake (so naïve, I know). During the article search process, I was already experiencing difficulties as many articles did not fully connect to the theories. After finally finding a suitable article, I came to the conclusion that ChatGPT has another very nice treat (not). When asking for citations from the article, ChatGPT liked to generate fictional citations that suited my text in the best way possible. Really sweet of my companion, but a bit illegal.
As I was dedicating so much time and prompts to article search and correcting the model if it was wrong again, the system started to slow down. With every input from my side, the lag increased. At the end, I had to wait minutes for the answer to finally show up.
Little did I know what kind of surprise I would receive at the end, just before handing in the essay. When reading the reference list one more time, I saw really weird looking website links. Immediately, I got a very bad feeling which was confirmed shortly after searching the internet: some articles did not even exist.
Lessons Learned
Luckily, I had some time (about two hours) left to erase the AI-generated articles and search for new existing articles. The latter turned out to be impossible given the time frame, so I had to rewrite or erase text in order to present it as my own work. Fortunately, I submitted the assignment in time and received a good grade for it
But the most important thing I received was a better understanding of ChatGPT’s capabilities and limitations. I actually overestimated its capabilities and faced the consequences of it. I spent way to much time on superficial interaction and the correction of the GenAI application, which was clearly visible in the interaction document I had to submit along with the final essay. The interaction document was basically a copy/paste of the entire interaction with the GenAI model. The length of my document was 140+ pages!
For the second essay, I tried a different approach because of this experience. Before prompting anything, I did my own research and searched for articles myself. I did use ChatGPT for understanding theory and later on for formatting, spelling and grammar correction. Remarkably, I received a lower grade for this essay but I did not really care as I saved much time. I had significantly reduced the number of interaction pages, thereby also reducing my frustrations and headaches.