We started working on AskCanvas, an AI assistant that provides course-grounded support for students and class-level insights for instructors, with one goal: make it the best we possibly could. After weeks of working on the project, I assumed our final recommendation would be “roll it out”. It turned out that I was wrong, and the reason was that our results looked too good to trust.
Starting with a clean split
My assigned parts were the governance and failure analysis and the final recommendation. However, by the time we had submitted, the line between my parts and everyone else’s had basically disappeared. After working on our parts individually, we ran into a few problems; most of our parts relied on another person’s part to be completed, and we weren’t sure about the coherence of our message overall. So we decided to get together and finish out the project as a team in the final days. This turned out to be a great decision not only for our project but personally. I really enjoyed being challenged on my parts as well as challenging my teammates on their assumptions and points. This had the added benefit of giving me a more well-rounded view of our project: from the financials and pricing strategy to the prototype and testing.
Decisions we changed
That shift in how we worked directly changed several of our decisions. The first was strategic. We had framed the main long-term threat as students taking their questions to AI tools outside of Canvas. After a lively debate, we realized students are neither our customers nor our competitors. We found that the bigger risk was competing platforms, such as Blackboard, copying the idea, so we rewrote the strategy around deeper integration with Canvas workflows. The next two changes stemmed from the responsible AI analysis that I wrote. First, when any questions unanswered by the AI assistant escalated to the Canvas discussion board and were answered, they would be automatically added to the AI’s database for future questions. However, upon discussion, our team agreed that it would be safest and more in line with the EU AI Act as well as the human-in-the-loop principle if the questions were only added to the database after instructor approval. Finally, we originally had questions directly tied to students with a toggle allowing for the questions to be anonymous. Again, keeping GDPR in mind, as well as recognizing that one point we were trying to solve was allowing more timid students to be included in the feedback process, we decided to make all questions anonymous by default.
When the results were better than expected
Designing AskCanvas was one thing. Testing it was where I got surprised. I expected the AI to struggle, and I assumed the final test would fail by missing patterns. Instead, it found all three intended patterns and correctly separated the noise. Its one mistake went the other way: it over-segmented, splitting one issue into two insights. My first reaction was that this was a pretty good problem to have, but I didn’t trust the result. The test had used 30 synthetic questions, and our manual baseline was simulated, not observed. Therefore, the 56.7% reduction meant fewer items to read, not proven time saved. And an output that looks confident and is mostly right is the “false confidence” risk I had written about in the governance section. A teammate’s early test, which produced two unsupported clusters, had already shown it. So instead of “roll it out,” we recommended a controlled pilot.
Feedback shaped our thinking as well. For instance, our professor pushed us to change the question we were testing, from “would professors like more feedback from their students?” to “can AI compile that feedback efficiently and accurately?”. That is a much harder claim to prove, and our evidence only partly supports it. The emails from various EUR staff didn’t change our direction, but they confirmed the problem and added a condition: the solution must not burden lecturers further. That condition is why our stop criteria say not to scale if the dashboard creates more work.
Looking back, I also see a limit in how we tested. If I were to do it again, I’d run the test with a real professor from the course, not a proxy.
What I learned from working as a team
Our group communicated well, and I learned that I love debating ideas, both my own and other people’s. But I’m better at debating than at deciding when to stop, and as a group we procrastinated more than we should have. Having my sections challenged also made them better, and challenging my teammates made me care about parts I hadn’t been assigned. What I’d change, though, is the order. We split the tasks first and only came together once the dependencies became a problem. Next time I’d start by collaborating, so we agree on the message and expectations before dividing the work. I’d also start testing earlier. My recommendation couldn’t be written until we had the results, so when testing ran long, we were left with very little time at the end to draft our final recommendation.
What I’d tell my past self
Before this project, I assumed, wrongly, that you should adopt new technology, AI included, as soon as it comes out. Now I start with the problem and only then ask whether GenAI is the best tool to solve it. In AskCanvas’s case, the problem was that instructors couldn’t see where students were struggling, and GenAI is a good tool for this because the student input is messy, unstructured questions. In addition to this, I learned the AI itself isn’t the advantage in our case, since competitors can use the same models.
The concept that stuck with me most was platform pricing: subsidizing the side that creates value. The ladies’ night example from class made me realize I use platforms and encounter this exact concept every day without thinking about them this way. We even used it ourselves. Students and instructors use AskCanvas at no cost to them; universities pay because the students’ questions are what create the value instructors get.
Writing the governance section also changed how I see my own AI use. In daily life, it’s easy to forget who has access to my data and why. On this project, the GDPR and the AI Act directly shaped the product: anonymous questions by default, and no answer entering the knowledge base without instructor approval. That is why I think the EU’s role in keeping AI safe is indispensable.
So when someone pitches me an AI product now, I ask two things: does it solve a real problem, and does it solve it better than the alternative? For AskCanvas, we had good evidence for the first and only partial evidence for the second, since our manual baseline was simulated. That is why my first instinct, “roll it out”, was wrong.
