On MareNostrum 4 there are several combinations of Compilers and MPI implementations available for the users. With this report we want to analyse the performance of the different combinations using the OSU Benchmarks. OSU offers the possibility of measure the performance of different MPI functions, as peer-to-peer or collectives and with different message sizes.

Combination analysed​

We named the combinations used with the following pattern:

• [compiler-module]-[mpi-module]

Where the [compiler-module] is the compiler module loaded and the [mpi-module] is the MPI implementation module loaded.

The different combinations that we have used are the following:

• intel2017.1-impi2017.1
• intel2017.4-impi2017.4
• intel2017.6-impi2017.6
• intel2017.7-impi2017.7
• intel2018.0-impi2018.0
• intel2018.1-impi2018.1
• intel2018.2-impi2018.2
• intel2018.3-impi2018.3
• intel2018.4-impi2018.4
• gcc7.2.0-openmpi1.10.4-apps
• gcc7.1.0-openmpi1.10.7
• gcc7.2.0-openmpi2.0.2
• gcc7.2.0-openmpi2.1.3
• gcc7.2.0-openmpi3.0.0
• gcc7.2.0-openmpi3.1.1
• gcc8.1.0-openmpi4.0.0
• gcc8.1.0-openmpi4.0.1
• gcc8.1.0-openmpi4.0.2
• gcc9.2.0-openmpi4.1.0
• gcc11.2.0_binutils-openmpi4.1.2
• intel2017.4_mvapich2.3b
• gcc7.2.0_mvapich2.3b
• intel2017.4_mvapich2.3rc1

Benchmarks​

The OSU Micro Benchmarks are a common suite of benchmarks for measuring and evaluating the performance of MPI operations for comparing different MPI implementations and the underlying network interconnect. We have used this suite to measure different MPI operations as Point-to-Point, collectives and non-blocking collectives.

Point-to-Point Benchmarks​

• osu_bibw - Bidirectional Bandwidth Test
• osu_bw - Bandwidth Test
• osu_mbw_mr - Multiple Bandwidth / Message Rate Test
• osu_latency - Latency Test
• osu_latency_mp - Multi-process Latency Test
• osu_latency_mt - Multi-threaded Latency Test
• osu_multi_lat - Multi-pair Latency Test

Collective MPI Benchmarks​

• osu_allgather - MPI_Allgather Latency Test
• osu_allreduce - MPI_Allreduce Latency Test
• osu_alltoall - MPI_Alltoall Latency Test
• osu_barrier - MPI_Barrier Latency Test
• osu_bcast - MPI_Bcast Latency Test
• osu_gather - MPI_Gather Latency Test
• osu_reduce - MPI_Reduce Latency Test
• osu_reduce_scatter - MPI_Reduce_scatter Latency Test
• osu_scatter - MPI_Scatter Latency Test

Non-Blocking Collective MPI Benchmarks​

• osu_iallgather - MPI_Iallgather Latency Test
• osu_iallreduce - MPI_Iallreduce Latency Test
• osu_ialltoall - MPI_Ialltoall Latency Test
• osu_ibarrier - MPI_Ibarrier Latency Test
• osu_ibcast - MPI_Ibcast Latency Test
• osu_igather - MPI_Igather Latency Test
• osu_ireduce - MPI_Ireduce Latency Test
• osu_iscatter - MPI_Iscatter Latency Test

Analysis​

Point-to-Point​

From the results below, we observe that some modules have bad performance on any P2P benchmark with any size. These modules are, intel2017.1-impi2017.1 and the the OpenMPI 4.X compiled with GCC.

For the Bandwidth benchmarks with small message, we observe better performance with the OpenMPI >= 3. In general, for the rest of the sizes, the IntelMPI, MVAPICH and OpenMPI <=3 have similar results in terms of bandwidth.

For the P2P Latency benchmarks, we observe similar performance using IntelMPI, OpenMPI and MVAPICH, except for the IntelMPI 2017.1 and the OpenMPI 4.0.X.

Collectives​

From the results below, we observe a similar behaviour using the OpenMPI 4.0.X versions, where the performance is deficient compared with the other combinations. In the following paragraphs, when we talk about OpenMPI, we do not include these 4.0.X versions with low performance. Also, we do not include IntelMPI 2017.1 when we speak about IntelMPI in general.

Specifically for each benchmark, we observe that for the AllGather the best option is using MVAPICH, then the OpenMPI different from 4.0.X and the Intel MPI, in this order. We observe the same for the Reduce operation.

On the AllReduce tests, we observe that MVAPICH is still the best, but in this case, IntelMPI has lower latency than the other OpenMPI. We observe the same on the Scatter function.

On the AllToAll, we observe similar performance for MVAPICH, OpenMPI and IntelMPI, with some variability. In this case, the OpenMPI 4.0.X runs did not finish for the larger message sizes.

On the BroadCast function, the MVAPICH and the OpenMPI are the best options except for the largest message size, where the MVAPICH is better than OpenMPI. In his case, the Intel MPI performs similarly to the OpenMPI 4.0.X.

On the Gather benchmark, the MVAPICH with small messages works a bit worse than IntelMPI and OpenMPI, but with the medium and large sizes, the performance is better using MVAPICH.

The Reduce_Scatter has similar behaviour to Alltoall where the three combinations perform similarly.

Non-blocking collective

For the Iallreduce, we observe that the OpenMPI 1.X have more latency for larger sizes.

For the Ibroadcast, we observe similar performance with IntelMPI, OpenMPI and MVAPICH.

For the Igather and Iallreduce, we generally observe better performance with the IntelMPI and MVAPICH, and a bit worse with OpenMPI.

For the Ireduce, we observe better performance in general with the OpenMPI, and similar performance if we compare IntelMPI and MVAPICH.

On the Iscatter operation, we observe better performance for the small size of IntelMPI and MVAPICH compared with OpenMPI, but for the larger sizes, the OpenMPI combinations have lower latency.

Conclusions​

To sum up, with the results observed, we do not recommend using on MareNostrum 4 the IntelMPI 2017.1 module and the OpenMPI 4.0.X modules, for any MPI application.

Then, in the case of an application with only P2P operations, we recommend using any MPI except the previously commented. If you use small message sizes, you can use OpenMPI 3 or below, but the improvement will be meagre compared with other MPI.

In the case of the Collective and Non-Blocking Collectives operations, we observe very different behaviours depending on the MPI function used, the message size and the number of nodes. We recommend checking the results from this report and the most consuming MPI functions of your application and choosing the best combination possible.

Results​

Point-to-Point​

osu_bibw​

osu_bw​

osu_latency​

osu_latency_mp​

osu_latency_mt​

osu_mbw_mr​

osu_multi_lat​

Collective MPI Results​

osu_allgather​

osu_allreduce​

osu_alltoall​

osu_bcast​

osu_gather​

osu_reduce​

osu_reduce_scatter​

osu_scatter​

Non-Blocking Collectives​

osu_iallgather​

osu_iallreduce​

osu_ialltoall​

osu_ibcast​

osu_igather​

osu_ireduce​

osu_iscatter​