I got instantly fascinated by ChatGPT’s capabilities once I tried it out, and ever since, I’ve been finding new and new ways to use its skills. Now I would like to share one of the less common use cases I tried: AI grading, because the results really surprised me and also highlighted the precautions needed when using and trusting GenAI tools.
When I applied to a master’s programme a couple of months ago, I realized I needed to take an English language exam to get accepted. I chose IELTS certification, and as someone who has lived and worked in English in a foreign country for a couple of years now, I wasn’t too worried about passing the exam, until I opened some IELTS prep forums and saw that pretty much everyone says the writing part is so difficult that even native speakers struggle to get a good score. That was the moment I realized I needed to practise my English essay writing skills a little more than I had anticipated. But since I didn’t want to spend hundreds of euros on IELTS prep courses or teachers to check my essays, I decided to try ChatGPT as my scoring and mentoring buddy.
I gave ChatGPT the detailed scoring criteria for each part of the writing section and asked it to provide me with feedback, scores for each subsection, and an overall score for the entire writing part. I started my first essay with a relaxed attitude, thinking I was in good hands, since all ChatGPT had to do was check my English essay and compare it to the detailed requirements.
To my surprise, it gave me a very low score every single time (I wrote about 10–15 essays), even lower than my required passing score. Its feedback always felt a bit shallow, and no matter how hard I tried to implement its recommendations, my scores just kept going down. So when exam day arrived, I felt completely demotivated and thought there was no way I’d pass the writing section.
I took the exam anyway. To my even bigger surprise, my result turned out to be one score band higher than I needed and two bands higher than ChatGPT’s average. Later, I looked it up, and YouTube is full of similar examples where ChatGPT or other GenAI models give lower scores than human examiners. One example that really stood out to me was a video by an official IELTS expert named Asiya. She tested AI scoring on three essays: her own, her fellow tutor Kevin’s, and her student Maria’s. The results turned out to be quite interesting: Asiya and Kevin both got slightly higher scores from the human examiner compared to the AI tools, but Maria’s essay was rated significantly lower by the AI than by the human.
So why do I think this was an important lesson? The obvious answer is what everyone keeps repeating: we need to use GenAI carefully and apply a critical mindset when considering what it says, especially if we’re not experts in the area. Secondly, it was interesting to notice how AI scoring, and simply knowing I was scored by AI, affected me. It made me feel genuinely demotivated, like I was fighting against a system whose opinion simply couldn’t be changed.
Still, I believe GenAI can be a very useful tool for (self)education, but probably in a hybrid setup, where it’s used for scoring together with a human, and the human ultimately has the final say on the result.
Also I would be very curious about other students’ experiences: Have you ever been scored/graded by AI? If yes, how did it feel? If not, would you be open to it?
It was interesting for me to read your post. Personally I can totally relate to the mixed feeling of frustration and fascination when using AI for learning purposes. AI tools can be very helpful, but it can also make you feel demotivated when it gives you inconsistent feedback.
I really like how you mentioned the emotional side of being graded by AI. I’ve also felt like “fighting against a system” sometimes when I used AI as a grading tool. I agree that even when we know that AI is just a tool, its judgment can still influence the way how we feel about our own work. I’ve had similar kind of experiences where AI gave me good tips, but its scoring just didn’t match what an actual teacher would have said.
I used ChatGPT as my IELTS test preparation coach, too, but the situation I encountered is quite different from yours. After I put down the prompt of how to grade, based on the IELTS official website, it scored me really fairly from my point of view. Sometimes my score is lower because I am not familiar with the topic of the question, while other times I am scored higher. My final result was somewhere in the middle of the range ChatPGT scored me.
I agree that critical thinking is a crucial skill when it comes to using GenAI. Do not believe in everything GenAI generates because it may be hallucinating or biased. However, using a correct prompt is also important at the same time. GenAI is not human, the way of asking or inputting a request should be different from simply talking to a human being. Another point is that when ChatGPT scores you lower is actually not a bad thing. I assume that you would rather be overprepared than underprepared!
As a life long English speaker I have always questioned the utility of tools like ChatGPT, they can indeed add a lot of value but if you just rely on them, you might run into some issues. As a TA, I would love to see AI be able to grade something automatically but your post illustrates the issue. It tends to be wrong. Depending on how you prompt it, you will either see a positive or negative answer, these models are trained to be in agreement with what we write unless we specify other wise. Even then, one can make the argument that it is doing what we want by going against us.
I really liked your post, it even gave me the inspiration to write about my own experience with AI and how it is useful but only to a point and comes with some downsides.
On a final note, I too have been overly influenced by how ChatGPT or another LLM rates my work (for it to end up being wrong.)
Finally, I saw some suggestions online about training writing with AI, and I used a similar approach to have ChatGPT give me revision suggestions and scoring criteria. After roughly two weeks of training, I nervously took the exam, because the scores AI gave to my writing samples were consistently between 6.0 and 6.5. But to my great surprise, I ended up getting a 7. Therefore, I am very grateful for the help AI provided me in writing practice.