Interactive Bodo Cluster Setup using Ipyparallel

Bodo can be used with ipyparallel to allow interactive code execution on a remote cluster.

Prerequisites

  • Install Bodo on every host (see Installing Bodo Community Edition).

  • Install ipyparallel in your Bodo conda environment:

    conda install ipyparallel -c conda-forge
    
  • Create an MPI profile for IPython:

    ipython profile create --parallel --profile=mpi
    

Starting an ipyparallel cluster

For a detailed guide on how to start an ipyparallel cluster, please refer here.

Controller and engines on the same host

Run the following command to start a cluster on the same host:

ipcluster start -n N --profile=mpi

# N: total number of engines to launch in your cluster

Multiple hosts

  • If you have engines on different hosts you have to set up passwordless SSH between each of these hosts (this is needed for mpiexec).

  • The controller must be able to connect to all engines via TCP on any port. If you have a restricted network, please refer to the ipyparallel documentation for other options such as SSH tunneling.

  • Start the controller (the controller can run on a different host than the engines):

    ipcontroller --profile mpi --ip '*'
    
  • Copy the ipcontroller-engine.json generated by the controller to each host that will run engines. This file is located in ~/.ipython/profile_mpi/security/ipcontroller-engine.json (copy to the same directory on the engine hosts).

  • Start the engines with mpiexec (run this command on one of the hosts in your cluster):

    mpiexec -n N -machinefile machinefile python -m ipyparallel.engine --mpi --profile-dir ~/.ipython/profile_mpi --cluster-id ''
    
    # N: total number of engines to launch in your cluster
    # machinefile: constains list of IP addresses or host names where you want to launch engines.
    

    Note

    Make sure your machinefile is in the following format:

    ip_1:N_1
    ip_2:N_2
    ...
    

    where N_i is the number of engines that you want to run on host i. This is necessary for mpiexec to map MPI ranks to hosts in consecutive order.

If you want to start a remote cluster with ipcluster please refer to the ipyparallel documentation.

Verifying your setup

To verify that your cluster is set up correctly, you can run the following code.

import ipyparallel as ipp

def test_MPI():
    comm = MPI.COMM_WORLD
    return "Hello World from rank {} on host {}. total ranks={}".format(comm.Get_rank(), MPI.Get_processor_name(), comm.Get_size())

def main():
    client = ipp.Client(profile='mpi')
    dview = client[:]
    dview.execute("from mpi4py import MPI")
    ar = dview.apply(test_MPI)
    result = ar.get()
    for out in result:
        print(out)
    client.close()

main()

On a cluster with two hosts running 4 engines, the correct output is:

Hello World from rank 0 on host A. total ranks=4
Hello World from rank 1 on host A. total ranks=4
Hello World from rank 2 on host B. total ranks=4
Hello World from rank 3 on host B. total ranks=4

Running Bodo code on your ipyparallel cluster

Before running Bodo code on your cluster, ensure that the cluster is running correctly and MPI-enabled. Please refer to the above section on how to do this.

Here’s an example of Monte Carlo Pi calculation with Bodo.

import ipyparallel as ipp
import inspect
import bodo

@bodo.jit
def calc_pi(n):
    t1 = time.time()
    x = 2 * np.random.ranf(n) - 1
    y = 2 * np.random.ranf(n) - 1
    pi = 4 * np.sum(x ** 2 + y ** 2 < 1) / n
    print("Execution time:", time.time() - t1, "\nresult:", pi)
    return pi
  • We define a Python wrapper for calc_pi called bodo_exec which will be sent to the engines to compute. This wrapper will call the Bodo function on the engines, collect the result and send it back to the client.

    def bodo_exec(points):
        return calc_pi(points)
    
  • We can send the source code to be executed at the engines, using the execute method of ipyparallel’s DirectView object. After the imports and code definitions are sent to the engines, the computation is started by actually calling the calc_pi function (now defined on the engines) and returning the result to the client.

    def main():
        client = ipp.Client(profile='mpi')
        dview = client[:]
    
        # remote code execution: import required modules on engines
        dview.execute("import numpy as np")
        dview.execute("import bodo")
        dview.execute("import time")
    
        # send code of Bodo functions to engines
        bodo_funcs = [calc_pi]
        for f in bodo_funcs:
            # get source code of Bodo function
            f_src = inspect.getsource(f)
            # execute the source code thus defining the function on engines
            dview.execute(f_src).get()
    
        points = 200000000
        ar = dview.apply(bodo_exec, points)
        result = ar.get()
        print("Result is", result)
    
        client.close()
    
    main()