There are many factors to consider when deciding whether to buy or build your AI monitoring system. Ultimately, it comes down to whether you can accomplish the outcomes you set out to achieve with ML monitoring, at a cost you can afford, and in a timely manner.
When It Makes Sense To Build
1. Your AI Program Is Immature; Your Needs Are Simple/Basic
The need to monitor production AI often arises early, frequently before the company deployed its first ML models in the business. Data science teams universally understand the importance of visibility into data integrity and model fidelity over time, with the need for optimizations and improvements after deployment. However, in some cases, when you're still mainly experimenting or using only manual and offline processes, perhaps there is not a lot of data complexity, and perhaps the level of adoption in the business is still rather low, so the stakes are not very high. In these cases, these teams may just need a simple dashboard with basic alerting, and open-source tools such as Grafana or Kibana can address these needs at this point in time.