Machine Learning Model Degradation

In a very insightful article made by David Talby he discuss about the fact that in a second that a Machine Learning goes to production, actually this model starts degradate itself because the model contact with the reality, where the author uses the following statement:

The key is that, in contrast to a calculator, your ML system does interact with the real world. If you’re using ML to predict demand and pricing for your grocery store, you’d better consider this week’s weather, the upcoming national holiday and what your competitor across the street is doing. If you’re designing clothes or recommending music, you’d better follow opinion-makers, celebrities and current events. If you’re using AI for auto-trading, bidding for online ads or video gaming, you must constantly adapt to what everyone else is doing.

The takeaways from the article it is that every ML in production should have some simple guidelines for monitoring and reassessment like 1) A online measure of accuracy to monitor the model degradation, 2) The ML Engineers most mind the gap between the distributions in the training and test sets, and 3) Data quality alerts regarding the unexpected growth in some groups of your sample that you’re facing some bad predictions.