Project Learning Blog – CostSentinel workflow and dashboard

My part of CostSentinel was the problem definition, the proposed workflow and the dashboard. I worked on explaining what a controller would see when a job started to look unusual and what the person could do with that information. One of the more difficult parts was deciding how much the interface could reasonably claim. It had to make the prototype understandable while being clear about what we had actually tested.
We focused on job shops and make-to-order manufacturers where costs can build up differently across jobs. The problem described in our proposal was that overruns might only become apparent during monthly reporting, after the job had already shipped. There is an obvious appeal and a real opportunity in giving someone an earlier warning. However, our simulation did not establish how often manufacturers experience that problem. The final deck sets out the workflow we want to use, which we still need to test in a pilot. I had to keep that qualification in the problem statement. The close-process benchmark was useful background, but it couldn’t tell us whether our assumed reporting delay was realistic for the simulated plant. The people involved also had different reasons to care. A controller would be looking at costs and margins. A plant manager would be better placed to investigate what was happening on the floor. The CFO would need to judge whether the service was worth paying for. I couldn’t describe all of them as a single user with the same needs. For the dashboard, this meant showing the cost element behind a deviation and a possible check, rather than just displaying a warning. The proposed response still depended on someone who understood the job.
After I was looking more closely at the model score I did realise that its label could make the product seem more certain than it really was. A high score looks persuasive on a screen so it would be easy to read it as the probability that a job will finish over budget, although our model was not calibrated to support that claim. I found that surprising because the number itself looks so straightforward. The interface therefore calls it a model score and explains that two consecutive daily scores above the threshold trigger an alert. I had to be comfortable showing the result with that limitation. Making the label shorter would not have made it more accurate.
The job replay helped explain why the timing of a warning matters. A person can move through a selected job and compare recorded cost with the time phased plan. The alert timeline makes it possible to inspect what happened at different points.
However, these are 13 selected synthetic examples that can be examined retrospectively. They do not show a live plant or establish how well a manager would use the product. I think this is easy to forget when looking at a working interface. We have a dashboard to show for our work, but its appearance cannot make up for the user evidence we haven’t collected. There was a similar issue with the explanation panel. The prototype uses templates, and generative AI is proposed for later development. A template can identify a cost element and suggest an investigation without knowing why the cost changed. An increase in material costs, for example, can justify checking purchasing conditions. It does not establish that those conditions caused the problem. I needed to keep that difference clear when describing the explanation feature. Otherwise, the interface would imply a level of understanding that the prototype does not have.
The evaluation also affected how I thought about the review process. CostSentinel detected 74% of overrun jobs before closure in the assumed base case, while 56% of emitted alerts were classified as valid. For the person using the product, that leaves a considerable amount of review work that does not result in a valid catch. It made the option to record an alert as real, benign or unsure more important to the workflow. The buttons demonstrate how a verdict could be captured. They do not demonstrate that verdicts improve the model, and the proposed process requires actions and final outcomes to be considered before retraining. I also had to be careful with the word “fixable”. The evaluation counts detections before job close. Some of the remaining expenditure may already be committed, so an open job is not necessarily one where costs can still be reduced. The economics uses an assumption about how much remaining driver cost could be avoided. In the product description, I was explaining early detection without presenting it as evidence of savings. I needed the meaning of the technical result as well as its headline percentage. I would now ask for that explanation before using a teammate’s figure in my own work.
There are still questions I can’t answer from the finished prototype. For example we didn’t see real managers decide whether to trust a warning, how much time to spend investigating it or whether to act on it. We also haven’t established whether the suggested checks would help them in their daily work. The proposed pilot is where these questions would have to be examined. I can use the prototype to explain the workflow, although I cannot claim that I demonstrated its usefulness to a real user.
I would be more cautious about describing an AI product as valuable just because it produces an earlier prediction. In CostSentinel, a warning still has to reach the right person while something worthwhile can be done. The controller needs enough information to decide whether to investigate, and the plant manager needs a practical opportunity to respond. That is what I want the pilot to examine. I believe that after we can see that a warning lead to a useful decision, it would give us evidence that the dashboard improves the process, independent of our own judgement of how it looks and works.