Recommended Cluster Configuration¶
This section describes best practices for configuring compute clusters for Bodo applications.
Minimizing Communication Overheads¶
Communication across cores is usually the largest overhead in parallel applications including Bodo. To minimize it:
For a given number of physical cores, use fewer large nodes with high core count rather than many small nodes with a low core count.
This ensures that more cross core communication happens inside nodes. For example, a cluster with two
c5n.18xlargeAWS instances will generally perform better than a cluster with four
c5n.9xlargeinstances, even though the two options have equivalent cost and compute power.
Use node types that support high bandwidth networking.
AWS instance types with
nin their name, such as
r5n.24xlargeprovide high bandwidth. On Azure, use virtual machines that support Accelerated Networking.
Use instance types that support RDMA networking.
Examples of such instance types are Elastic Fabric Adapter (EFA) (AWS) and Infiniband (Azure). In our empirical testing, we found that EFA can significantly accelerate inter-node communication during expensive operations such as shuffle (which is used in join, groupby, sorting and others).
Ensure that the server nodes are located physically close to each other.
For most applications, we recommend using
c5n.18xlarge instances on
AWS for best performance. For memory intensive use cases
instances are a good alternative. Both instance types support 100 Gbps
networking as well as EFA.
Other Best Practices¶
Ensure that the file descriptor limit (
ulimit -n) is set to a large number like
This is especially useful when using IPyParallel which opens direct connections between
Avoid unnecessary threading inside the application since it can conflict with MPI parallelism.
You can set the following environment variables in your shell (e.g. in
bashrc) to avoid threading: