This section details the ownership of the simulation, model, and evaluation aspects of CostSentinel, an early-warning system developed to forecast budget overruns in manufacturing jobs. Primary responsibility in this instance is verifying the numbers generated during the data generation stage, as well as assessing comparative performance and the legitimacy of the assessment underpinning those results, since some figures may be used in slide headers and dashboards.
Establishment of the baseline. Without using genuine factory data, 1,200 synthesised jobs were created during the evaluation stage, each exceeding budget by more than 10%. Inputs to the simulator represented assumptions about the delays, errors, and frequencies of root causes of overruns. All of this has been done prior to obtaining results, with the help of a colleague who has experience working in Europe and during an internship at Accenture Strategy. Importantly, these inputs have not been calibrated to ERP data.
Observation monitoring
The realization is that observation monitoring only picks up 10% of overruns. On the other hand, the monthly report produced through simulation picks up 29.5% of overruns prior to closure, almost three times more effective than this monitoring method. Therefore, the baseline favors the manual monitoring, but the model’s presumed superiority may be underestimated rather than overestimated.
Design criterion
A design guideline stipulates that at least half of the warnings ought to be true. This criterion was considered key to the entire proposition, since if most alerts were false, operational controllers would ignore this tool.
Accuracy metrics assessment. Typical market talk surrounding machine learning (ML) systems emphasises the need for accuracy above 90%. However, in this instance, 26% of jobs exceed the budget limit. If the tool gives no alerts, its accuracy is already 74%. Thus, traditional metrics do not provide the insights we need here. Precision and recall were used as evaluation metrics for this model, followed by an audit of the process before acceptance of the evaluation.
Fixing data leakage
The first problem was leakage: jobs from year one could carry over into year two, and cross-year group labelling disrupted the training procedure. The split was fixed by limiting the training-only jobs to those closed by the cutoff date and to postings made before the testing period; as a result, 77 open jobs have been excluded. Furthermore, cost curves have been fit within the training fold, ensuring that validation jobs did not influence the curve used to assess them.
Ensuring the fairness of the baseline. The second problem concerned a comparison with some straightforward solutions. The original baseline uses a fixed budget threshold (1.15×) that a single snapshot on a given day can trigger. Conversely, the model’s threshold is selected based on validation data and requires two consecutive days of alerts. This makes the comparison unfair, since we are measuring the ability to tune rather than the intrinsic performance. The correction has been made accordingly. Rules and the model are now on equal grounds. Both need two consecutive days and implement a 5-day cool-down period after the alert is dismissed. Both threshold values are selected based on validation data; as a result, the model’s lead will necessarily decrease, but I continue to do it this way.
Interpretation of the revised figures
Given the current parameters, the algorithm identifies 74% of running jobs with 56% accuracy before termination, which meets the desired minimum criteria. The month-end report identifies 30% of the running jobs with 36% accuracy. The time-phased rule is the simplest and most efficient, identifying 51% of running jobs with 14% accuracy. No simple rule can identify 50% of the jobs.
This means that the model not only increases the catch rate but also decreases the noise level, producing 3.8 alerts per week, whereas the phased rule produces 20 alerts. Even though this benefit is simulated, it cannot be considered proof of cost savings.
Unexpected finding: The clean data, which was expected to perform best given the absence of data issues, captured the largest number of overruns (98%). However, its precision on test data is only 45%, and it successfully passed the validation stage at 53%, vs the more complex baseline scenario, which achieved 56% precision.
This does not mean “Better data makes poorer performance”. Each scenario underwent independent retraining and recalibration, and even the thresholds differ (Clean uses 0.45, and Baseline 0.95). The model deployment cannot be based solely on the baseline results, because another scenario might not meet the criteria.
Three limitations existed in interpreting the results. First, the score obtained by the model is not probabilistic. This is because the score denotes the sum of two probabilities: the probability of overrun and the probability of deviating from the curve. In other words, a score of 0.95 does not imply a probability of 95%. Second, while the simulated controller was aware of the jobs’ costs, the actual controller was not. Therefore, the report focused on determining whether an overrun would occur. Third, only one synthetic plant and only one seed were considered in the experiment.
Collaboration
After the model finished training, the project was shared on GitHub, allowing my teammates to run it independently using their own mock data. They could also train the model on their own synthesised data, which would increase confidence in its soundness.
This belief was based not on the realism of the results but on the mere ability to reproduce them. The crucial point is that the difference has always been clear to all of us. My teammates needed not just the updated numbers but also the threshold value and a precise definition of the alert. The same issue applies to the dashboard, which has thirteen job replays. This only demonstrates the alert procedure, while precision metrics can be obtained only with analysis of the full test cohort.
This changed my understanding of AI and strategy in general. The precision is a strategic variable, not just a metric of the model. Every false alarm consumes some time of the controller. Thus, the 50% floor expresses a question of trust – how much noise can the team endure before it becomes unwilling to use the tool?
There are two ideas underlying such a statement. First of all, the technological solution is valuable only when it can alter the workflow system: capability, workflow, and value. Our model represents the first stage, while the controller review procedure represents the second. Precision determines whether the value is generated. Secondly, information becomes valuable only when it is credible, diagnostic, and decision-relevant. A score of 0.95 is not a 95% probability, and the simulated manager knows the job’s final costs in advance. Thus, the alerts should be considered unverified until a real back-test on the actual dataset has proved their effectiveness.
In other words, the headline claim of 90% accuracy does not seem particularly relevant, given that most jobs do not overrun their budgets. Thus, it has changed our recommendation from “Deploy” to “Back-test on real ERP data and Pilot”.
Now, before accepting the AI product as correct, I try to check three things: whether an alternative has been given a fair chance to show its performance; whether test data were actually absent during training and who bears the costs of model mistakes.
It is important that the final costs are taken into account and that any necessary adjustments are made quickly. Basic guidelines for comparison remain the same. There is one problem that needs to be resolved: the amount of manual analysis required when a warning appears in the ERP extract.
