MareNostrum 5
System overview
MareNostrum 5 is a pre-exascale EuroHPC supercomputer supplied by Bull SAS that combines Lenovo ThinkSystem SD650 V3 and Eviden BullSequana XH3000 architectures, providing two partitions with different technical characteristics.
MareNostrum 5 GPP (General Purpose Partition)
The MareNostrum 5 GPP, a general-purpose system, houses 6,408 nodes based on Intel Sapphire Rapids (4th Generation Intel Xeon Scalable Processors), along with an additional 72 nodes featuring Intel Sapphire Rapids HBM (High Bandwidth Memory).
This configuration results in a total count of 725,760 processor cores and 1.82TB of main memory, as arranged in the table below:
| Node type | Node count | Cores per node | Main memory per node |
|---|---|---|---|
| GPP | 6,192 | 112 | 256GB |
| GPP-HighMem | 216 | 112 | 1,024GB |
| GPP-HBM | 72 | 112 | 128GB |
GPP/GPP-HighMem nodes are mainly equipped with the following:
- 2x Intel Xeon Platinum 8480+ 56C 2GHz
- 16x DIMM 16GB/64GB 4800MHz DDR5
- 960GB NVMe local storage
- ConnectX-7 NDR200 InfiniBand (shared by two nodes, 100Gb/s bandwidth per node)
Underlying topology of a GPP node
Additionally, GPP-HBM nodes are provided with:
- 2x Intel Xeon CPU Max 9480 56C 1.9GHz
- 8x Die 16GB 3200MHz HBM2
- 960GB NVMe local storage
- ConnectX-7 NDR200 InfiniBand (shared by two nodes, 100Gb/s bandwidth per node)
Underlying topology of a GPP-HBM node
MareNostrum 5 ACC (Accelerated Partition)
The MareNostrum 5 ACC accelerated block comprises 1,120 nodes based on Intel Xeon Sapphire Rapids processors and NVIDIA Hopper GPUs, offering a total (CPUs + GPUs) of 680,960 compute units.
ACC nodes are chiefly equipped with the following:
- 2x Intel Xeon Platinum 8460Y+ 40C 2.3GHz (80 cores per node)
- 4x NVIDIA Hopper H100 64GB HBM2
- 16x DIMM 32GB 4800MHz DDR5 (512GB main memory per node)
- 480GB NVMe local storage
- 4x ConnectX-7 NDR200 InfiniBand (800Gb/s bandwidth per node)
Underlying topology of an ACC node
Infiniband connection
The MareNostrum 5 infiniband topology is a three-layer fat-tree with a total of 324 switches (model QM9790). It has an implementation of 3 GPP islands, 1 storage island and 7 ACC islands. The layer levels are as follows:
Layer 1: the switches that are directed to the nodes
Layer 2: connects the node switches to the switch cores
Layer 3: the core switches
There are two types of configuration: advanced routing and adaptive routing. Up/Down routing with fault tolerance is used.