Deploying Bodo with Kubernetes¶
This section demonstrates an example showing how to deploy a Bodo application with Kubernetes. We deploy Bodo with the Kubeflow MPI-Operator, which enables resiliency in the case of Node Failure for long running Bodo applications.
You need the following to deploy your Bodo application using Kubernetes:
Access to a Kubernetes cluster.
For this example, we'll use AWS EKS. See the section below on creating an EKS cluster to see how we set it up.
A Docker image containing the Bodo application scripts and their intended Bodo version made available on a Docker registry, so that Kubernetes can pull it.
pi.py: can be used to validate your setup.
chicago_crimes.py: can be used to test a use-case with Bodo on a larger dataset.
You can use this as a base image for your own Docker image. If you want to use a private registry, you can follow the instructions here.
Make sure to provide the correct CPU and Memory requests in the Helm or YAML file for your Bodo jobs. If correct values are not provided or the cluster doesn't have sufficient CPU or Memory required for the job, the job will be terminated and worker pods may keep respawning. You can get a good estimate of the CPU and Memory requirements by extrapolation from running the job locally on a smaller dataset.
Creating an EKS Cluster using KOPS¶
Here are the steps create an AWS EKS cluster using KOPS.
Install KOPS on your local machine:
Create a location to store your cluster configuration:
First you need to create an S3 bucket to use as your
Create your cluster:
The following code block creates a cluster of 2 nodes each with 4 cores . You can modify the
node-countargument to change the number of instances. To change the number of worker nodes, update
node-size. You can deploy the cluster in a different AWS region and availability zone by modifying the
master-sizerefers to the leader that manages K8s but doesn’t do any Bodo computation, so you should keep the instance size small.
Finish creating the cluster with the following command.
This might take several minutes to finish.
Verify that the cluster setup is finished by running:
Deploying Bodo on a Kubernetes Cluster Using Helm¶
You can deploy Bodo applications on a Kubernetes cluster using Helm by following the steps outlined below.
Clone the Bodo examples git repository.
Set up Helm and run your application.
First, check if Helm is already installed in your system usingthis guide to set up Helm in your system. To run the example Bodo application, navigate to the repository and run the following command
To run the chart with your job specification, navigate to the Kubernetes/helm folder, edit the
values.yamlfile, and run the above command.
Retrieve the results.
When the job finishes running, your launcher pod will change its status to
completed, and any information published to stdout can be found in the logs of the launcher pod:
- If you want to remove the job, run
- If you want to delete the MPI-Operator CRD, run the command
Deploying Bodo on a Kubernetes Cluster Manually¶
Install MPIJob Custom Resource Definitions(CRD)
You can check whether the MPI Job custom resource is installed via:
The output should include
mpijobs.kubeflow.orglike the following:
Run your Bodo application.
spec.slotsPerWorkerwith the number of physical cores (not vCPUs) on each node
spec.mpiReplicaSpecs.Worker.replicasto the number of worker nodes in your cluster.
- Build the image using the Dockerfile or use
bodoaidocker/kube-mpioperator-minimaland replace the image at
- Check the container arguments is referring to the python file you have intended to run
- Lastly, make sure
-nis equal to
spec.slotsPerWorker, i.e. the total number of physical cores on your worker nodes.
Run the example by deploying it in your cluster with
kubectl create -f mpijob.yaml. This should add 1 pod to each worker and a launcher pod to your master node.
- View the generated pods by this deployment with
kubectl get pods. You may inspect any logs by looking at the individual pod's logs.
Retrieve the Results.
When the job finishes running, your launcher pod will change its status to completed and any stdout information can be found in the logs of the launcher pod:
- When a job has finished running, you can remove it by running
kubectl delete -f mpijob.yaml.
- If you want to delete the MPI-Operator crd, please follow the steps on the MPI-Operator Github repository.