Artificial Intelligence (AI) is a very relevant trend and has already been discussed several times in the first weeks of Information Strategy. One of the most popular examples of generative AI nowadays is ChatGPT. Shortly after its release, I used ChatGPT in December 2022 for the first time. At the time, I was pursuing the pre-master at RSM before starting BIM, and we had a course called Research Training that made use of R.
I was very interested and had never used something similar before, so I was looking for something I could feed it. Luckily, some of the content of the Research Training course was unclear to me: the perfect time to put it to the test. It was already known that you should ask ChatGPT specific questions in order to retrieve the most useful answers. Therefore, I copied and slightly adjusted the readability of the exercise into the website. There was an exercise that gave seven observations (7, 15, 26, 25, 7, 1, 5). The student had to determine the Z-score of these seven numbers. This has to be calculated using the formula Z = (x-μ)/σ. I was surprised by the extensiveness of ChatGPT’s answer (see picture below that was made in December). Not only did it provide me with the answer, it gave me a step-by-step guide on how to answer the question. I was blown away!
But then… 1.52 was not the correct answer. How was that possible? It just guided me through all the steps to land here? I read through the steps again but this time, I used my calculator to check its calculations. It turned out ChatGPT was not as smart as I thought. Out of all the relatively difficult tasks it had to think of, it calculated the mean incorrectly; the easiest of them all! I copied the question again and ran the question another couple of times. It gave me different results every time. All in all, a slightly disappointing, yet funny, outcome of the first use of ChatGPT.
AI needs a lot of practice and input in order to improve itself. Therefore, I tried the exercise again in October, almost a year after its first release. Although the setup of the answer was improved, the answer was still wrong.
In conclusion, ChatGPT is still fairly new and this is only one of its many great functionalities. After all, it made a relatively small mistake in this question. However, the input is clear to me and looking at the answer, ChatGPT does understand it. Therefore, the question to me remains how it still arrives at the wrong conclusion.
Hi Emma,
Thank you for you blog. I am intrigued by your findings. While we are all using ChatGPT as the AI-driven tool that “has all the answers”, this is an eye-opener that we should not believe everything the tool claims…. However, since your example was of 2022, is it still relevant? Or has the tool been improved?
I am surprised that, while ChatGPT has been a significant subject in the news, this hasnt got to my attention before. Furthermore, I believe it is a good reminder that we should stay critical!