Have you ever woken up at 3 a.m. and wondered if ChatGPT always has the right answers?

No? Well, me neither. But I thought it would be a good topic to write about so buckle up. Let’s first set some ground rules. This is a 300-word blog to prove the most advanced generative AI in the world (as of this writing) wrong. So you do the math, I won’t be conducting any advanced experiments here. However, this does not mean I will take the easy route either. I am not going to ask ChatGPT to guess my mood or anything subjective. To be more scientific, I will ask ChatGPT 2 questions (geography and math). I will share all the responses below. With that sorted out, here are the questions I will be testing:

• What countries start with V?
• I’ve got six cards that have the numbers one to six on them. I’m going to shuffle them and then deal them out from left to right. What’s the probability that the cards alternate between odd and even numbers (either starting with an odd number or an even number, then switching between odd and even for each subsequent card)?

Below are my findings. I hope you enjoy reading them.

1. For this question, there are 4 countries that start with V. And as you can see, ChatGPT did get it right (although not right off the bat). But you must first clarify that you want all the countries that start with V. If not specified, ChatGPT will give you the most (un)common answers to your question.

2. Now this algebra question was taken from Oxford’s 2022 Undergraduate Mathematics Admissions Test (Question 10, University of Oxford, 2022). So while it is difficult (for me anyways), it is not overly advanced as it was meant for high school students entering university. The answer we are looking for is 10% and interestingly, ChatGPT did get this question wrong. After several nudges toward the right path, it did eventually get it right (as you can see below).

A small investigation did reveal why ChatGPT got it wrong, and this time it is not based on the clarity of the prompt (see below).

Instead, the errors in the calculation were made from an assumption of not considering the arrangements of odd and even numbers.

From these two questions, we can see ChatGPT’s shortcomings and reasons for each failure. The first one is the clarity of the response and the second is the pre-assumptions that ChatGPT has which influences its way of answering a question. It should be a reminder that ChatGPT is meant as a tool to assist students in academia and not replace teachers yet. But most importantly, students should not easily use ChatGPT as a sole source of knowledge when studying. When asking questions on ChatGPT, it is best to have some basic knowledge of the area so students can grasp the accuracy of ChatGPT’s answers because, as proven here, it is not always correct! Thank you for reading my blog I hope you learned something new from here(:

References:

• University of Oxford. (2022, July 30). Solutions. Solutions | Mathematical Institute. https://www.maths.ox.ac.uk/study-here/undergraduate-study/maths-admissions-test/mat-livestream/algebra/solutions