Passenger A-0143’s father needs surgery the following day at 7 AM. As per KLM’s rule-based rebooking logic, Passenger A-0143 is booked on a flight arriving the following day at 12:30, which is too late. In this case, our AI system identifies the message from the passenger and books her on a flight arriving the same evening. The above example was an illustration on one of the slides of our presentation, and I feel it captured the essence of the entire project in one instance.
My role
The concept of doing flight rebooking came from myself. It was something that I thought of while preparing for the first sessions and then brought to the team. Then, my teammates refined it further to make it more concrete and realistic. Once we settled on the concept, I developed the project specification and prepared our datasets for the simulation. As we did not have access to actual data from KLM, we needed to create our 550 fictional passengers, who would travel by three different cancelled flights, namely New York, Nairobi, and London. I expected the coding to be the hard part, but it was the easy one. What was difficult was understanding the actual data structure of an airline that includes connections, frequent flier status, groups, and special needs, and reflecting it in our dataset. I also needed to prepare the messages for passengers, even some fake urgent ones. The simulation was initiated by Oskar, while Nathan and I completed it. Finally, I completed the simulation runs and produced the final results. In my bachelor’s thesis I worked with data and machine learning, but there the correct answers were given. Here I had to write the answer key myself, and deciding what a “correct” rebooking is proved harder than building the agent, which matches Agrawal et al.’s (2018) point that you must define how success is measured before AI can help.
Decisions and iterations
In the selection of this option over others, which was, for example, company-specific artificial intelligence training, we chose the rebooking agent because it was concrete enough to benchmark. The former rebooking option entailed the development of an agent that would rebook all passengers. However, after we discovered that there already existed some rule engines used by airlines to rebook 90 percent of the affected passengers, we shifted our attention to passengers who are not suited for the automatic approach, for example, a family whose bookings were separate and one that had deadline information contained in its message. I had assumed AI would be most valuable doing the whole job, but the rule engines already handle routine cases well. I now think AI pays off in the exceptions: decisions where the outcome depends on information a rule cannot read, such as a deadline hidden in a passenger’s message (Agrawal et al., 2018). In the first simulation run, we received surprising results. The outcome was “Stop” as per our pre-defined pass marks, meaning that on the Nairobi route, the agent did not perform any better than the rule engine. We examined the failure areas and came up with general solutions such as providing the passenger with the earliest deadline rather than the cheapest flight. In the second version, the agent outperformed the rule engine on all three routes. Outperforming the rule engine, however, was not the same as passing our own test.
What surprised me
There were two things that surprised me the most. First, passengers are much safer than I imagined. They receive compensation up to €600 and food as well as hotels under EU261 (Regulation (EC) No 261/2004, 2004, Art. 7). Second, it turned out that standard rebookings are already automated, and therefore the application of AI is less wide in scope and narrower than I had thought. Moreover, I expected faster rebookings to cut compensation costs. The simulation showed no difference, because EU261 compensation depends on how late the passenger arrives, not on how quickly the rebooking is processed. This changed what I mean by “value”: our agent does not save KLM money, it gets the right passenger onto the right flight. That benefit is harder to put in a business case, even though it is the one passengers notice most.
How evidence changed the work
At first I saw the pass marks as a formality. Setting them before the first run turned out to be crucial: when the first result came back as “Stop”, we could not quietly move the goalposts. Kohavi and Thomke (2017) argue that firms should decide on data rather than opinion, and I now understand this only works if the bar is fixed before you see the data. We had to make an exception once when we could not achieve the target, but we mentioned that in the change log as well. Overall, the agent rebooked 91–95% of passengers correctly, compared with 88–92% for the rule engine. Our pass mark asked for more than beating the baseline, and the agent cleared it on only one of the three routes. This is why we advised against implementation and suggested a shadow pilot, where the agent makes suggestions and staff make the final decisions, reflecting the split between prediction and judgment (Agrawal et al., 2018; Wilson & Daugherty, 2024).
Collaboration
Our team was fresh, and things started off a little chaotic at first. We still did not know each other’s strengths, and it took some time to coordinate ourselves. I expected four motivated people to organise themselves, but in the first week several of us drafted problem definitions in parallel while nobody had started on the dataset. But after some time, we were coordinating perfectly well. We distributed tasks among us: I was responsible for project definition, dataset, and simulations; Oskar for process flows and economics; Nathan for simulations, organization, and outputs; and Jules for regulation. I realized that teamwork functions best if everyone owns something specific.
How my view of AI and strategy changed
Since I knew the technical part, the main lessons learned were in the domain of business. I am still sure that the future belongs to AI agents and that soon they will manage all the rebooking and make this process smoother for passengers. However, our results showed that being slightly better than a rule-based system is not enough to replace it. The Agentic Future Test changed how I see strategy: I expected the agent itself to be KLM’s advantage, but once passengers and competitors have their own agents, it stops being a differentiator. As Barney and Reeves (2024) argue, AI rarely creates a new advantage but can amplify an existing one, which for KLM means its data, rules and seats. Demo Day reinforced this: the most convincing projects were those tested against a realistic baseline and honest about where they fell short.
In this sense, I see the same pattern in my personal work. Technical aspects of my work will become more and more automated, while knowledge of business, metrics, and knowing when to say “not yet” will remain crucial.
