When a flight is cancelled and many passengers are in need of a new seat, KLM’s passenger recovery system automatically searches for the first available option. What these recovery systems do not consider, however, are the constraints and needs of the passengers. For the AI Strategy Lab, my team built a simulation in which an AI agent reads passengers’ messages and passes the information on to those existing rule engines, which then assign the seats. My role during this project was Builder, as I was the one building the simulation with the help of Claude. In this reflection, I will cover three moments that shaped how I think about AI and strategy, and how these moments influenced our project.
Finding out about KLM’s passenger recovery system
Our initial idea was to build an AI agent for KLM that would rebook each passenger after a flight got cancelled. However, during a team meeting I found something that would change our course of action. In a Dutch court ruling between KLM and several stranded passengers, KLM mentioned that they also use a certain system for automatic rebooking of passengers on cancelled flights, called OPR (Optimizer Passenger Recovery) (Gerechtshof Amsterdam, 2022, para. 4.9).
We had to decide how we wanted to change our approach and so we discussed several options. One option was to move to a budget airline, that, as far as we knew, had no such system in place. I then suggested building on the existing system and trying to improve its results, as I thought that might be more feasible. Rather than building something disruptive, we could actually improve the current rule engine by adding an AI agent layer on top of it. After some discussion, in which we continued to refine both options, we decided to continue with building on the existing system. I then wrote to the teaching team on behalf of the group. Their feedback mostly confirmed that we were on the right track, as their suggestions aligned with what we had in mind.
Looking back at the project, I believe this strategic choice has been the most crucial. What convinced me was the reading of Furr and Shipilov (2019), who argue that successful digital transformation often comes from adaptation rather than reinvention. Rather than replacing the whole existing recovery system, our AI agent adds value in places where the current system has its limitations.
One message interpreted in three different ways
In my role as Builder, I worked in a fixed order that I learned specifically for this project, in which I explore the problem, agree on a plan together with Claude, let Claude write the code, and review the result myself. The planning phase is the most crucial, as in this phase I decide what the code should do before it is written.
After running the first version (v1) of the simulation, I saw that the verdict was Stop. I was quite surprised, as I expected a better verdict after spending considerable time on planning. However, the review showed some significant failure patterns. For example, the same message, which was a visa appointment taking place “tomorrow morning”, had been interpreted in three different ways. One got a deadline at 06:00, the other at 23:59 that night, and the last one did not receive a deadline at all. Same model, same message, but three deadlines. I had expected one single, structured answer, but the result reminded me that an AI model’s output is highly dependent on the quality of its instructions. If your instructions leave room for interpretation, the model will make sure to use it.
Although the verdict of v1 said Stop, we collectively decided that we wanted to revise. The identified failure patterns were fixable, which showed that the problem resulted from the agent’s instructions and not the core idea itself. Therefore, I started working on a second version (v2). As I performed my own analysis first, I deliberately asked Claude to analyse the failures and give suggestions for improvement itself. In that way, I could enhance my own analysis by adding details I had missed. While v2 gave the verdict Revise instead of the desired Pilot, it clearly outperformed the first version. To test the consistency of v2, I ran the same simulation again and saw that the messages had been interpreted similar to the first run. This test gave me evidence that the current version was consistent in the interpretation of passengers’ messages. This experience taught me that system reliability comes from the instructions and guidelines of a model, rather than the model itself. Because of this, we restricted the role of the AI agent strictly to reading the messages and left the actual seating allocation to the rule engine that is based on strict, hard-coded rules.
The decision not to build v3
As the Pilot verdict had not been met by v2, we were faced with another dilemma: building a third version (v3) or staying with the current verdict. Building v3 was possible, but it would have meant creating an entirely new database. Reaching the Pilot verdict in the existing database would only have been possible if we tweaked the instructions to match the specific passenger constraints. This was not desirable, as we only wanted to apply general rules. Therefore, we concluded that the Pilot verdict could only be achieved if we created a whole new database. This was not practical given the time we had left. Thus, we kept v2 as our final outcome and recommended a shadow pilot as the next step.
Two lessons on working in a team
This project has given me two major takeaways regarding collaboration. The first one is the need to remain flexible regarding roles. In the beginning, one of my teammates took on the role of Builder, but he hit a roadblock after his attempt to build the simulation with n8n did not work out. We discussed the matter and together decided to swap roles, as I was able to build the simulation using Claude. In an academic group project with unfamiliar teammates, you often discover each other’s capabilities only when the project progresses. In my opinion, it is therefore crucial to remain transparent and flexible, so roles can be reassigned based on the team’s set of abilities, or in our case who can leverage the right tools.
The second takeaway is to keep everyone involved in decision-making. Although everyone had their own role, we made all important decisions collectively. Changing the course of action after we found out about KLM’s OPR, revising after v1 gave the Stop verdict, and not building v3 were all decisions made by the whole team. In future group projects, I would like to implement these two takeaways proactively to maximise efficiency and to ensure that we can react quickly when facing setbacks.
While the project reinforced my belief that AI can have great impact when it augments existing solutions, it also proved that a good-looking output can still be inaccurate. Making an AI agent reliable requires a strict set of instructions and checks, as well as a clear distinction between what the agent reads and what the fixed rules decide. This follows the principle of separating prediction from judgment, as described in the AI Canvas (Agrawal et al., 2018). The AI agent predicts the needs of passengers, whereas the judgment call is made by the rule engine. That is also why our most important decision was to build on an existing solution that already works, rather than building something disruptive from scratch.
References
Agrawal, A., Gans, J., & Goldfarb, A. (2018). Prediction machines: The simple economics of artificial intelligence. Harvard Business Review Press.
Furr, N., & Shipilov, A. (2019). Digital doesn’t have to be disruptive. Harvard Business Review, 97(4). https://hbr.org/2019/07/digital-doesnt-have-to-be-disruptive
Gerechtshof Amsterdam. (2022, August 23). Judgment of 23 August 2022, case no. 200.286.212/01 (ECLI:NL:GHAMS:2022:2422). Rechtspraak.nl.
https://uitspraken.rechtspraak.nl/details?id=ECLI:NL:GHAMS:2022:2422
