Project Learning Blog

During our project, I mainly focused on the prototype, evaluation, and the governance part. Together with Thomas I made the prototype using ‘Lovable’, which is a prototype tool, and we worked on its measurement. I wrote the first versions of the evaluation and the governance and failure analysis. Next to that, I set up the measurement features in our Lovable prototype, prepared the Ruler publications for the test, reviewed the AI-generated packages myself, and presented the prototype together with Thomas on Demo Day. I also used an AI assistant throughout the project to check our draft against the rubric, and to write prompts for Lovable. That helped me work faster, however it also forced me to judge its output critically, because it often pointed out weaknesses in our work that I would have preferred not to see.
The biggest change in the project was the way we did the evaluation. Our first evaluation compared a rules-only pipeline with the AI pipeline on alert decisions. The numbers looked reasonable, but the two systems had been scored on different publications. Which meant the comparison said almost nothing. On top of that, the first review times in the app turned out to be around 12 seconds. Those were clicks on ‘approve’, and not real reviews. At that point I realised our evaluation was measuring the wrong thing and that our business case was about consultant time and quality, however we were testing screening accuracy.
After this, we decided to redesign a big part of our prototype. We compared AI assisted preparation to preparation done manually, as a consultant would. We used publications that the company Ruler had already once turned into alerts so we had a real quality reference. Then I added fields into the prototype for these references. The AI model was never able to see these references, so that it could not have a prejudice. This was very useful as we could now use the prototype for good measurements.
There are two things that surprised me about the AI itself. First, in our development round, the prototype gave a confident “no alert” on a DNB newsletter that should have been a human review. There was nothing also in the output that signalled doubt. Additionally, there was not a draft that could be used without edits. The AI was not wrong often, however it was pretty vague. It did name the correct regulator, topic and deadline, however it did not really name a lot of specifics a client needs. I expected errors to be the main risk. However, exclusion and omissions turned out to be more of an issue, as nothing in the draft looked wrong.
There were also things I did not do well. At the beginning, I was writing the evaluation while being pretty naive. I saw it mainly as a section we had to fill in and I accepted the first numbers from the prototype without really asking what they measured. When the review time came out at 12 seconds I did not really question it. I also did not fully understand what a baseline was for. Also I was not questioning if it mattered that the rules and the AI had been tested on different publications. Then I started to look back at the bigger picture and our business case, and I saw that our test did not really answer the question our business case depended on. From here on I rebuilt the prototype and I adjusted the way of measuring with a new baseline.
I thought the feedback we received on our draft was useful. It did point out that we had claimed things that our data did not show. An example is that the rules pipeline had a better performance than the AI pipeline. Another example is that we called the newsletter case a missed alert even though our own label said “needs review.” This made it more accurate for us to reframe it as a missed escalation. I also learned to separate three types of evidences in our prototype so we could compare it better.
In my opinion the collaboration went well. It really helped that we started on time so we had enough space and time to properly discuss things. Everyone took ownership of their own part which was very pleasant. Next time I would change some things in our communication. The coordination mostly happened through text messages. We mostly worked on the project in our own times, and because of this, I believe we all had a slightly different picture of the project at some points. A lot of text messages were sent while not everyone was working on it at that particular time. This resulted in us having to go through each other’s work to make sure everything was aligned. This led to some inconsistencies we had to change. Next time, I would plan more short group calls at key moments, and plan moments to work on the project with the whole team at the same place and time.
Before making this project I really saw AI as something to automate work and replace things. Now, I am starting to see AI as something that changes where human effort is needed. In our project and prototype, the consultant has to do less searching and drafting and the review becomes more important. The more important errors that the prototype makes are quiet errors, like mistakes that do not look wrong, such as a missing detail.
While making this project, my mind was also changed a bit about competitive advantage and AI. I think any rival is able to connect an AI language model to public regulatory articles. So the AI itself does not per se give an advantage. For Ruler, the advantage really comes from what the AI is combined with. Such as the legal frameworks and the trust of its clients. I also learned that it is not as easy as I thought to prove that a prototype creates value. It is even more difficult than just building the prototype. That is why our final recommendation is a three-month pilot while it uses Ruler’s own consultants.