Announcing the Launch of Enhanced Drift Detection in the Wallaroo ML Operations Center
We are pleased to announce the launch of our enhanced assay capabilities within the Wallaroo Model Operations Center.
Models require to be retrained from time to time as the parameters like historical data that was used to train the model changes. Data scientists need to identify when the model is drifting and generating unexpected predictions so they can retrain the model and understand the input parameters that contributed to the drift. For example, we work with customers with ML-based dynamic pricing models where, for example, changes in competitor offers can change local demand, such that the model needs to be updated to reflect the new environment.
Today, identifying this drift requires a lot of human effort through constant manual monitoring, making it harder for business to derive impactful insights. To identify drift you will have to compare the model predictions with actual results, as there are no automatic alerts, which makes early detection hard and leads to lag before you can retrain your model.
Most observability solutions that are available in the market today require you to download, transform, and move the output data to monitor and look at insights. This means an additional layer of operational overhead with data engineering required to get the data to a point where it’s model observability ready. This effort can take weeks or months — time that customers don’t have when data becomes highly volatile and models start to drift which can have serious business impacts.
Introducing Wallaroo’s Enhanced Assay Capabilities in the ML Operations Center
Wallaroo has built a model insights framework where you can create monitoring tasks called Assay. While this was previously available via our SDK, the ability to create and schedule Assays is now available via the Wallaroo Model Operations Center. This is integrated in the platform and runs with the pipelines so you get instant insights instead of downloading the data to monitor. This is especially helpful when there are a lot of modals and they are all running in production. Instead of a data scientist manually logging in to run a benchmark, the Assay will automatically show the drift, alerting the data scientist that they need to take a look. From there they can check if there are changes in parameters or if the Model needs to be retrained.
You can try our Community Edition for free and follow this step-by-step guide to set up your own model assays.
