Installing Wallaroo Enterprise in an Air-Gapped Environment

7 min readSep 21, 2022

There are many reasons why enterprises could be looking to deploy machine learning via air-gap. For example, the equipment could be far from internet connectivity at the edge, such as in oil derricks, gas pipelines, or in agriculture. Additionally, the increase in cybercrimes has led to some companies exploring the option of air-gapping to keep their systems secure. By isolating their networks from external ones, they can prevent the vulnerabilities that come with having these connections like data breaches and ransomware attacks that can cost billions in losses.

In recognition of this, Wallaroo has been designed to install and deploy even in air-gap environments, not only to ensure a company’s security, but also to help conserve bandwidth, deploy your models in remote settings (read more about our edge deployment architecture here), and demo them in the field without the need for internet connectivity.

In this tutorial, we will cover the process of installing Wallaroo into an existing Kubernetes cluster based on your network connections and system performance. It covers all the requirements and the process of downloading assets, installing Kots and the Kots Admin Console, and installing the Wallaroo Admin Console and Air-gap.

Prerequisite Conditions for Wallaroo Air-gap Installation

Before beginning a Wallaroo Air-gap installation, the following prerequisites must be met:

Background knowledge of Kubernetes and managing internal clusters
Installation of a Kubernetes cluster that has access to a private container registry as well as the push and read credentials for that registry
A jump box or other method of remote connection to the cluster to be adopted in the commands

Environment Hardware Requirements

For Wallaroo to function on a Kubernetes cloud cluster, the following minimum system requirements must be met:

Minimum of 4 Nodes, each with 16 GB RAM
Minimum of 625 GB of total storage, allocated to 5 users with up to 4 pipelines, each with 5 steps. Deployment of any additional pipelines will further demand 50 GB of storage per lab node.
A minimum of 8 CPU Cores are required for basic functionality, but 16 Cores are recommended to enable all services.

Enterprise Network Requirements

For the basic functionality of Wallaroo, the network requirements below must be met:

200 IP addresses per cloud environment for Wallaroo Enterprise users
98 IP addresses per cloud environment For Wallaroo Community users
DNS services integration for Wallaroo Enterprise edition for access to the supporting services of the Wallaroo instance.

Environment Software Requirements

The software requirements for Wallaroo can be categorized into:

Environment Requirements: A Kubernetes Version from 1.20 to 1.22 and runtime with containerd installed in the environment running the Wallaroo version.
Kubernetes Admin Requirements: Install the software in the Kubectl Tools and the Kots CLI where kubectl and kots are installed respectively, to manage the Kubernetes environment.

Node Selectors

Configure the following node selectors to allocate various services and resources to different node pools.

ML Engine Load Balance node selector
ML Engine node selector
Prometheus node selector
Database Node Selector
Grafana node selector
Nifi Node Selector
Each Lab * Node Selector

Installation Process of Wallaroo Enterprise in an Air-Gap Environment

Step 1: Downloading Assets

Begin by downloading the required assets via your organization’s License and Air-gap Download page as shown below:

You can also copy the links and enter the following commands into the node for Air-gap installation using curl:

Wallaroo Enterprise License File: downloaded as yaml

curl -LO {Link to Wallaroo Enterprise License File}

Wallaroo Air-gap Installation File: downloaded as wallaroo.airgap

curl -Lo wallaroo.airgap '{Wallaroo Airgap Installation File URL}'

KOTS CLI: downloaded as kots_linux_amd64.tar.gz

curl -LO {Link to KOTS CLI}

KOTS Air-gap Bundle: downloaded as kotsadm.tar.gz

curl -LO {Link to KOTS Airgap Bundle}

Save the files on the air-gap server that manages the Kubernetes cluster before proceeding to air-gap the clusters and installing the downloaded software.

Step 2: Installing Kots

To install kots on the node administering the Kubernetes cluster, first, extract the archive using the command below:

tar zxvf kots_linux_amd64.tar.gz kots

Next, enter the following command to install kots to the /usr/local/bin directory, modifying the path to correspond to the location of the kubectl command:

sudo mv kots /usr/local/bin/kubectl-kots

If the kots is successfully installed, the results should indicate the version as follows:

kubectl kots versionReplicated KOTS 1.81.0

Step 3: Installing the Kots Admin Console

Depending on your company’s registry setup, adjust the command below to extract the KOTS Admin Console container images, pushing them into a private registry with push access:

kubectl kots admin-console push-images {KOTS Airgap Bundle} \{Private Registry Host}:{Private Registry Port} \--registry-username {Registry Push Username} \--registry-password {Registry Push Password}

In the above command:

Private Registry Host: refers to the URL of the private registry host used by the Kubernetes cluster.
Private Registry Port: refers to the port of the private registry used by the Kubernetes cluster (default: 5000).
KOTS Air-gap Bundle (default: kotsadm.tar.gz): Downloaded with other assets in step 1.
Registry Push Username: refers to the username with push access to the private registry
Registry Push Password: refers to the corresponding password of the registry user with push access to the private registry.

Step 4: Installing Wallaroo Air-gap

Enter the command below to install the Wallaroo Air-gap file into the Kubernetes cluster using the Kots Admin images:

kubectl kots install wallaroo/ea \--kotsadm-registry {Private Registry Host}:{Private Registry Port} \--registry-username {Registry Read Username} --registry-password {Registry Read Password} \--airgap-bundle {Wallaroo Airgap Installation File} \--namespace {Wallaroo Namespace} \--license-file {Wallaroo License File}

Modify the above command using registry credentials with read access, automatically generated as an imagePullSecret on all of the Admin Console pods and utilized in pulling images. The credentials will be saved in a Kubernetes secret in the same namespace as the Admin Console, and will include:

Private Registry Host: refers to the URL of the private registry host used by the Kubernetes cluster
Private Registry Port: refers to the port of the private registry used by the Kubernetes cluster (default: 5000)
Wallaroo Namespace (default: wallaroo): refers to the Kubernetes namespace used to install the Wallaroo instance
Wallaroo Air-gap Installation File (default: wallaroo.airgap): Downloaded among other assets in Step 1
Wallaroo License File: Downloaded among other assets in Step 1
Registry Read Username: refers to the username with read access to the private registry
Registry Read Password: refers to the password of the registry user with read access to the private registry

You can speed up configuration by adding the following flags:

— shared-password {Wallaroo Admin Dashboard Password}: The Wallaroo Admin Dashboard password
— config-values config.yaml: configures the Wallaroo instance depending on the specified yaml file
— no-port-forward: Doesn’t forward port 8800 for use
— skip-preflights: Launches the Wallaroo instance without running pre-flight checks.

Unless the option — -no-port-forward is chosen, a connection to the Wallaroo Admin Console will be availed on completion as shown below:

Press Ctrl+C to exit
Go to http://localhost:8800 to access the Admin Console

Responding with Ctrl+C will disable the Wallaroo Admin Console, leaving the Wallaroo instance and services running in the cluster. Therefore, you can use the command below to re-enable the Admin Console:

kubectl-kots admin-console --namespace {Wallaroo Namespace}

Step 5: Preflight Checks

Unless the — skip-preflights flag was used, Wallaroo will run Preflight checks to ensure that the Wallaroo Instance is compatible with the required specifications.

Step 6: Wallaroo Admin Console

In case you did not provide the license file in the command line, you can still use it via the Wallaroo Admin Console accessible on port 8800. Configure the air-gapped cluster with a method of port forwarding through the jump box to access the Wallaroo Admin Console.

Step 7: Status Checks

You may need to troubleshoot or contact the Wallaroo support team if the status page is displayed as Missing or Unavailable for twenty minutes or more.

However, if the installer successfully completes allocating resources and deploying workloads, the status indicator will turn green and indicate Ready as shown below:

Step 8: Troubleshooting

Wallaroo’s Admin Console generates troubleshooting bundles for technical support, which entail logs and configuration files that can be evaluated before sending to Wallaroo. To exclude sensitive data, such as passwords, tokens, or other personal information, from the logs, the console also provides a configurable redaction option.

You can manage the support bundles simply by logging into the administration console and selecting the Troubleshoot tab, followed by the Analyze Wallaroo option. You can then save the file as a compressed archive by selecting Download bundle and sending it to the Wallaroo technical support. You can also review and download any existing bundle from the Troubleshoot tab at any moment.

To speak with one of our experts about installing our ML deployment solutions in your air-gapped environments, reach out and contact us. Please visit the Wallaroo documentation site for more information or assistance on this how-to.