When our team began this project, I expected it to be a primarily technical exercise. Coming from a technical background, I assumed the hard part would be to actually build something, and I would take away most knowledge from the construction process. What I have learned is almost the opposite. Building the prototype was easy, figuring out what to build is what taught me the most.
The first idea I had was SmartStock, an AI which would predict how much inventory a grocery store would need. It was a compelling idea, and I could imagine how I would build it, but upon further inspection and a talk with Prof. Ting Li, it had two problems. The idea was too broad, and modern stocking software already solves this issue. I learned that before even asking if something can be built, one has to ask if someone would want the product.
After that, my teammate proposed Crit Happens, an AI to help run D&D sessions when there isn’t a dungeon master. We met with our team to expand on the idea and decided what each of us would work on. I had the prototype and evaluation. The collaboration was easygoing in this project, and I enjoyed working on it. We kept everything in a shared document with no issues, and that’s why I will be using this strategy in the future for team projects.
My greatest contribution to the project was the clickable prototype, and I had a lot of fun making it. I had the idea of having the user pick their story, then pick a narrator, then build the party, ending with the chat itself. I also generated narrator portraits and let my teammates record voice greetings to give the prototype a feeling of a tangible product instead of just an empty shell.
In order to understand what features D&D players would want in our product, we watched several YouTube videos from people who played or explained the rules. This showed us what we should do and, more importantly, what we shouldn’t. Much of the premium plan came from what these videos told us.
Thinking about the evaluation was my other major contribution to the project, and I think what I learned will help me greatly in the future. As someone with a technical background, I am much more likely to ask if something works than if it works better than something else. With our evaluation, however, we set a system to see how much better our solution was than either a human or just standard ChatGPT. We also defined success criteria before building to avoid confirmation bias. We also admitted limitations that our data had a very small sample size, that there was a novelty effect we couldn’t account for. This project also made me rethink what AI can be, and I think the value of voice AI in particular will only grow. Before this project, I had little to no experience with conversational AI, and the project made me realize just what it can do. The voice interactions were surprisingly expressive and engaging, and I believe that voice AI can be a huge improvement for not just games, but many applications.
Our business model work also showed me the limitations of using those existing tools. Standard voice AI is already very capable, and freely available to anyone from different companies. However, a generic model is not necessarily a good Dungeon Master by default, since to perform well it has to be trained on the rulebooks and actual sessions in order to create a good setting and follow the rules. The model also has to have access to the character sheets and statistics, and any additional material relevant to the current session such as the adventure notes and the state of the current scene. Dice rolls, narrator voices and a human override then provide the illusion of a real campaign. This is the space that we would like to occupy, and as far as I know, there is no one else doing this, which would give us a niche to work with. Presenting the project at the demo day was another great experience for me. I appreciated the opportunity to see the audience interact with the prototype. It felt significantly better than working on the project alone behind a computer screen. All the visitors enjoyed the presentation, and their positive feedback was the result of hard work and dedication put into the project. The final product, featuring portraits of the narrators and voice-over narration appeared as a compelling piece of software. Moreover, the experience yet again demonstrated the importance of validating ideas with real people rather than focusing on internal discussions only.
This project taught me to value the business side of things. Before this, I thought strategy was something that happened around the work we did as developers. Now I see that it defines the work from the start and determines if development even makes sense. From scrapping SmartStock to listening to players to planning an honest evaluation, these lessons have taught me to think critically about the assumptions I make.
