How to Shadow Deploy Models with Wallaroo

5 min readSep 8, 2022

In regards to model testing and experimentation, Wallaroo offers several different frameworks for model evaluation against other models or sets of models inside a pipeline. This allows you to select which of the models you have available would perform better in production. Being able to test existing models against a potential replacement helps facilitate better decision-making from your data.

One type of testing framework is through parallel deployments, also referred to as “shadow deployments”. Single or multiple new models known as “challengers” are tested against an existing control model already known to generate accurate results called the “champion”, to determine if the “challenger” can generate faster or more precise outputs depending on the criteria. The key with shadow deployments is you get to see how a challenger would perform in the production environment without actually being in production (and possibly giving bad results to the business).

In this tutorial, we will illustrate how to deploy challenger models and compare their performance against the champion model using Wallaroo. This will cover a step-by-step process of uploading the champion and challenger models, creating a shadow deployment in a Wallaroo pipeline, performing an inference with a shadow deployment, observing the data and shadow_data results from the Inference Result Object, and viewing the pipeline logs and pipeline shadow logs.

Prerequisites for Shadow Deployment

For the sake of this guide, we shall be using the files below which are available via the Wallaroo tutorials repository.

Sample test data for the inference testing: dev_smoke_test.json
The champion model: keras_ccfraud.onnx
Challenger Models: modelA.onnx and xgboost_ccfraud.onnx

Step 1: Importing libraries

Start by importing the required libraries using the following command:

import wallaroofrom wallaroo.object import EntityNotFoundError

Step 2: Connecting to Wallaroo

Next, type in the command below to connect to your Wallaroo instance and save the connection as the variable wl:

wl = wallaroo.Client()

Step 3: Setting Variables

Depending on your organization’s requirements and Wallaroo instance, you will need to specify the following variables to create or use existing workspaces and pipelines, as well as upload models. For example, in this guide the model files were placed within a folder named “models” therefore the file locations are written ‘models/(file name)’. Begin with the following commands, making adjustments as necessary to meet your organizational needs:

workspace_name = ‘ccfraud-comparison-demo’pipeline_name = ‘cc-shadow’pipeline_name_multi = ‘cc-shadow-multi’champion_model_name = ‘ccfraud-lstm’champion_model_file = ‘models/keras_ccfraud.onnx’shadow_model_01_name = ‘ccfraud-xgb’shadow_model_01_file = ‘models/xgboost_ccfraud.onnx’shadow_model_02_name = ‘ccfraud-rf’shadow_model_02_file = ‘models/modelA.onnx’sample_data_file = ‘./smoke_test.json’

Step 4: Creating or Connecting to a Workspace and a Pipeline

It is not a requirement in the Wallaroo instance to have unique workspace and pipeline names. Therefore, use your organization’s standards to create or connect to a workplace and a pipeline by defining the workspace_name variable and the pipeline_name variable respectively in the command below:

def get_workspace(name):wl = wallaroo.Client()workspace = Nonefor ws in wl.list_workspaces():if ws.name() == name:workspace= wsif(workspace == None):workspace = wl.create_workspace(name)return workspacedef get_pipeline(name):wl = wallaroo.Client()try:pipeline = wl.pipelines_by_name(pipeline_name)[0]except EntityNotFoundError:pipeline = wl.build_pipeline(pipeline_name)return pipelineworkspace = get_workspace(workspace_name)wl.set_current_workspace(workspace)pipeline = get_pipeline(pipeline_name)pipeline

Step 5: Loading the Models

Having created and set your workspace as current in the previous command, your models will be uploaded and listed as champion, model2, and model3 according to the variable names assigned in step 3. The commands should mirror the following:

champion = wl.upload_model(champion_model_name, champion_model_file).configure()model2 = wl.upload_model(shadow_model_01_name, shadow_model_01_file).configure()model3 = wl.upload_model(shadow_model_02_name, shadow_model_02_file).configure()

Step 6: Creating Shadow Deployment

You can now proceed to create shadow deployment using the add_shadow_deploy(champion, challengers[]) command. In this command, champion refers to the primary model used for inferences run through the pipeline, while challengers[] denotes the sets of models used for inferences iteratively:

pipeline.add_shadow_deploy(champion, [model2, model3])pipeline.deploy()

It will generate the following results:

Step 7: Running Test Inference

A test inference will be created using data from sample_data_file.

The inference result data from the champion and the challengers[] models will be returned through the Inference Result Object’s data element and the Inference Result Object’s shadow_data element respectively:

response = pipeline.infer_from_file(sample_data_file)response

The response will look like this:

Step 8: Viewing Pipeline Logs

Once the inferences have been completed, employ the following pipeline logs function to extract the log data from the pipeline:

pipeline.logs()

If the process is successful, it will appear as follows:

Step 9: Viewing Shadow Deploy Pipeline Logs

The following pipeline logs_shadow_deploy() command will allow you to view the results and inputs for the shadow deployed models:

logs = pipeline.logs_shadow_deploy()logs

The logs from a shadow pipeline, grouped by their input will be as shown:

Step 10: Undeploying the Pipeline

Lastly, once you’ve completed your shadow deployment and received the pipeline logs, undeploy the pipeline to restore the resources to the system. The command for this process is:

pipeline.undeploy()