Intro​

Since VASP is one of the most used applications in MareNostrum 4 and given the imminent arrival of MareNostrum 5, which will incorporate GPUs, we have tested VASP6 with CPUs and GPUs to understand how it behaves and which is the expected performance we can see depending on their input.

One of the critical steps of any HPC program is reading data and storing temporal or final results to or from the file system. Knowing how and when to read or write into a file is essential because it is one of the most time-consuming operations. Not only is it vital to be efficient when doing IO operations (input-output), but it is also crucial to how the data is accessed and how to keep it. Another critical decision is how many processors will execute these IO operations and how they will communicate. In this case, these tests have been executed on gpfs/projects on Marenostrum 4. We will use a pure MPI communication scheme and HDF5 1.10.1.

Considering this, we have performed several tests using the h5bench benchmark to see the performance and impact of different memory access patterns and temporary or end-file storage patterns using a set of data structures.

On MareNostrum 4 there are several combinations of Compilers and MPI implementations available for the users. With this report we want to analyse the performance of the different combinations using the OSU Benchmarks. OSU offers the possibility of measure the performance of different MPI functions, as peer-to-peer or collectives and with different message sizes.

Combination analysed​

We named the combinations used with the following pattern:

• [compiler-module]-[mpi-module]

Where the [compiler-module] is the compiler module loaded and the [mpi-module] is the MPI implementation module loaded.

The different combinations that we have used are the following:

• intel2017.1-impi2017.1
• intel2017.4-impi2017.4
• intel2017.6-impi2017.6
• intel2017.7-impi2017.7
• intel2018.0-impi2018.0
• intel2018.1-impi2018.1
• intel2018.2-impi2018.2
• intel2018.3-impi2018.3
• intel2018.4-impi2018.4
• gcc7.2.0-openmpi1.10.4-apps
• gcc7.1.0-openmpi1.10.7
• gcc7.2.0-openmpi2.0.2
• gcc7.2.0-openmpi2.1.3
• gcc7.2.0-openmpi3.0.0
• gcc7.2.0-openmpi3.1.1
• gcc8.1.0-openmpi4.0.0
• gcc8.1.0-openmpi4.0.1
• gcc8.1.0-openmpi4.0.2
• gcc9.2.0-openmpi4.1.0
• gcc11.2.0_binutils-openmpi4.1.2
• intel2017.4_mvapich2.3b
• gcc7.2.0_mvapich2.3b
• intel2017.4_mvapich2.3rc1

Benchmarks​

The OSU Micro Benchmarks are a common suite of benchmarks for measuring and evaluating the performance of MPI operations for comparing different MPI implementations and the underlying network interconnect. We have used this suite to measure different MPI operations as Point-to-Point, collectives and non-blocking collectives.

Point-to-Point Benchmarks​

• osu_bibw - Bidirectional Bandwidth Test
• osu_bw - Bandwidth Test
• osu_mbw_mr - Multiple Bandwidth / Message Rate Test
• osu_latency - Latency Test
• osu_latency_mp - Multi-process Latency Test
• osu_latency_mt - Multi-threaded Latency Test
• osu_multi_lat - Multi-pair Latency Test

Collective MPI Benchmarks​

• osu_allgather - MPI_Allgather Latency Test
• osu_allreduce - MPI_Allreduce Latency Test
• osu_alltoall - MPI_Alltoall Latency Test
• osu_barrier - MPI_Barrier Latency Test
• osu_bcast - MPI_Bcast Latency Test
• osu_gather - MPI_Gather Latency Test
• osu_reduce - MPI_Reduce Latency Test
• osu_reduce_scatter - MPI_Reduce_scatter Latency Test
• osu_scatter - MPI_Scatter Latency Test

Non-Blocking Collective MPI Benchmarks​

• osu_iallgather - MPI_Iallgather Latency Test
• osu_iallreduce - MPI_Iallreduce Latency Test
• osu_ialltoall - MPI_Ialltoall Latency Test
• osu_ibarrier - MPI_Ibarrier Latency Test
• osu_ibcast - MPI_Ibcast Latency Test
• osu_igather - MPI_Igather Latency Test
• osu_ireduce - MPI_Ireduce Latency Test
• osu_iscatter - MPI_Iscatter Latency Test

Analysis​

Point-to-Point​

From the results below, we observe that some modules have bad performance on any P2P benchmark with any size. These modules are, intel2017.1-impi2017.1 and the the OpenMPI 4.X compiled with GCC.

For the Bandwidth benchmarks with small message, we observe better performance with the OpenMPI >= 3. In general, for the rest of the sizes, the IntelMPI, MVAPICH and OpenMPI <=3 have similar results in terms of bandwidth.

For the P2P Latency benchmarks, we observe similar performance using IntelMPI, OpenMPI and MVAPICH, except for the IntelMPI 2017.1 and the OpenMPI 4.0.X.

Collectives​

From the results below, we observe a similar behaviour using the OpenMPI 4.0.X versions, where the performance is deficient compared with the other combinations. In the following paragraphs, when we talk about OpenMPI, we do not include these 4.0.X versions with low performance. Also, we do not include IntelMPI 2017.1 when we speak about IntelMPI in general.

Specifically for each benchmark, we observe that for the AllGather the best option is using MVAPICH, then the OpenMPI different from 4.0.X and the Intel MPI, in this order. We observe the same for the Reduce operation.

On the AllReduce tests, we observe that MVAPICH is still the best, but in this case, IntelMPI has lower latency than the other OpenMPI. We observe the same on the Scatter function.

On the AllToAll, we observe similar performance for MVAPICH, OpenMPI and IntelMPI, with some variability. In this case, the OpenMPI 4.0.X runs did not finish for the larger message sizes.

On the BroadCast function, the MVAPICH and the OpenMPI are the best options except for the largest message size, where the MVAPICH is better than OpenMPI. In his case, the Intel MPI performs similarly to the OpenMPI 4.0.X.

On the Gather benchmark, the MVAPICH with small messages works a bit worse than IntelMPI and OpenMPI, but with the medium and large sizes, the performance is better using MVAPICH.

The Reduce_Scatter has similar behaviour to Alltoall where the three combinations perform similarly.

Non-blocking collective

For the Iallreduce, we observe that the OpenMPI 1.X have more latency for larger sizes.

For the Ibroadcast, we observe similar performance with IntelMPI, OpenMPI and MVAPICH.

For the Igather and Iallreduce, we generally observe better performance with the IntelMPI and MVAPICH, and a bit worse with OpenMPI.

For the Ireduce, we observe better performance in general with the OpenMPI, and similar performance if we compare IntelMPI and MVAPICH.

On the Iscatter operation, we observe better performance for the small size of IntelMPI and MVAPICH compared with OpenMPI, but for the larger sizes, the OpenMPI combinations have lower latency.

Conclusions​

To sum up, with the results observed, we do not recommend using on MareNostrum 4 the IntelMPI 2017.1 module and the OpenMPI 4.0.X modules, for any MPI application.

Then, in the case of an application with only P2P operations, we recommend using any MPI except the previously commented. If you use small message sizes, you can use OpenMPI 3 or below, but the improvement will be meagre compared with other MPI.

In the case of the Collective and Non-Blocking Collectives operations, we observe very different behaviours depending on the MPI function used, the message size and the number of nodes. We recommend checking the results from this report and the most consuming MPI functions of your application and choosing the best combination possible.

Results​

Point-to-Point​

osu_bibw​

osu_bw​

osu_latency​

osu_latency_mp​

osu_latency_mt​

osu_mbw_mr​

osu_multi_lat​

Collective MPI Results​

osu_allgather​

osu_allreduce​

osu_alltoall​

osu_bcast​

osu_gather​

osu_reduce​

osu_reduce_scatter​

osu_scatter​

Non-Blocking Collectives​

osu_iallgather​

osu_iallreduce​

osu_ialltoall​

osu_ibcast​

osu_igather​

osu_ireduce​

osu_iscatter​

During the execution of synthetic and real app benchmarks, we detected that the performance of the benchmarks have a high variability on MareNostrum. Also, we were notified by other colleagues that they achieved more stable times with other apps, as FALL3D, on clusters like JUWELS. Therefore, we decided to investigate the origin of this “system noise” and the different ways to limit it, as its affects the stability of the application performance and affects negatively in the scalability, mainly in applications with synchronization MPI calls in their kernel.

Using a synthetic benchmark developed by us, we observed that there are periodic events during the simulation that affects the time of a small number of iterations. We concluded that these events come from the system and they produce preemptions on the simulation. To avoid these preemptions, we tried to run the same benchmarks leaving 1 or 2 cores per node empty. Thus, these 1 or 2 empty cores are available to run these periodic events therefore the other cores avoid the possible preemption and the applications running on them could have a more stable time per iteration.

In addition, during the PRACE UEABS activity, we noticed that some applications perform better on SkyLake systems with HyperThreading enabled, as JUWELS cluster (beyond the performance improvement acquired by the higher frequencies). This is because the systems with HyperThreading handle better the preemption and the context switching are swifter.

Taking these points into account, we decided to test multiple applications with multiple configurations, with HyperThreading and without limiting the frequency to 2,1 GHz, as others SkyLake clusters do.

On MareNostrum 4, the frequency is limited to 2,1 GHz by a recommendation of the vendor. During some tests removing this limit, we detected that depending on the application the performance increases or decreases. Also, we observed on some cases a higher power consumption.

Therefore, we run several tests measuring the performance and power consumption on nodes with and without the frequency limit.