My role and contribution
Within Team 34, my contribution focused mainly on the technical side of the project. I designed the n8n
workflow that we ultimately recommended, built the clickable prototype of the “COR support AI-agent”
chatbot that simulates this workflow, designed and ran the evaluation we used to benchmark and
iteratively improved the prototype. In practice, this meant more than simply ‘translating’ requirements
into technology. It involved identifying business requirements, determining which aspects of the
business model could be optimized, defining the respective strategic roles of humans and technology
within the organization, and selecting the technology best suited to address the identified needs. Every
branch in the workflow, from identification and the support agent to the Jira escalation and the closing
of a session, represents a decision about where AI ends and human judgment begins.
Important decisions and iterations
The first version of the prototype was not really about COR. It started as a generic logistics and
consulting assistant, as most large language models (LLMs) are trained to give generic answers when no
data is provided. Once the team had established that costs, efficiency and availability were key to the
value proposition, and once we had learned from a COR employee how product inquiries are handled,
we had defined the requirements for the prototype. COR’s clients are agencies and consultancy
companies that use a project-management platform and need software support, not consulting advice.
That reframing changed the knowledge base, the tone of the replies and the kinds of questions that
should be escalated. For example, the AI agent’s LLM used keyword referencing to identify tasks, which
hurt performance. That is why I proposed using a different type of vectorization for the agent’s context
window, which worked exceptionally well and, according to online sources, even outperformed the
industry standard.
Thinking about the bot as if it were actually deployed changed how I looked at security. In the prototype
everything runs on mock data, so a mistake costs nothing. In production, however, the chat window is
public, every message costs tokens and the agent handles sensitive client data. That is why the workflow
starts with a landing page and an identity check, and uses a message limit per session and a ticket limit
per month to reduce spam and prompt injection.
I also chose to keep the prototype as a single HTML file with a side panel showing which n8n nodes ran
and what the session record looks like. This let teammates and reviewers without a technical background
follow the workflow logic themselves instead of having to trust a diagram.
What surprised me or failed
The biggest surprise was how fragile simple keyword matching is. The prototype’s LLM uses keyword
matching instead of semantic search, and this produced collisions I had not anticipated: “sla” matched
“Slack”, “employee” threatened to hijack questions about COR’s employee-management features, and
“how much” sent statistics questions such as “how much time did we spend…” to the pricing escalation.
Even the order of the data in the sample database mattered: “billable hours” was mistaken for “hours”
until I moved the statistics data to the top. This reminded me to do further research on how these
models actually work under the hood.
How feedback and evidence changed the work
Rather than simply claiming that the bot works, I made a test set of 44 customer questions and labelled
the expected outcome of each before running them: 30 how-to and statistics questions, 10 that need a
person and 4 that are out of scope. The prototype handled 38 of the 44 correctly on the first message
(86%), compared with an industry first-contact resolution (FCR) of 65% for technical support. Most
misses came from keyword matching: vocabulary gaps (“vacation days”, “meetings”), keyword order,
and one confident wrong answer where “write a proposal” was misread as project setup. The last case
worried me most, because a confidently wrong answer damages a customer’s trust not only in the AI
agent, but also in the whole company. An end-to-end test also exposed a defect where, after a failed
login, the next user was greeted with the previous person’s name.
What I learned from collaboration
Working with teammates who were developing the business model and had more anecdotal information
about the company forced me to reverse my way of solving problems: instead of searching for a problem
that technology can solve, I had to start from the problem itself. Since they had a different background
from mine and knew more about the industry than I did, I realised that my initial proposals during
brainstorms did not actually solve a problem. For example, simply using AI to send follow-up emails and
summarise data is only a small part of a project manager’s work and returns little financial benefit,
whereas enabling these people to handle more projects, provide accurate data and focus on specific
tasks addresses a real problem that this AI agent solves. Also, working entirely independently is not a
good idea, as some parts (such as governance principles) need technical expertise combined with
practical knowledge to identify properly.
How the project changed my understanding of AI and business strategy
Before this project I tended to see AI as the product: choose a capable model and the value will follow. I
now see the model as the most replaceable part of the system. COR’s competitive advantage lies in its
own knowledge base and the feedback loop from real customer questions, which will keep improving
over time. This is why we argue that domain knowledge is more important than any AI model. AI agents are a commodity that every large-scale business can access, but obtaining valuable data will become an area of competition in the age of AI.
