The recently completed European-funded project provides innovative solutions for adaptative storage, specifically designed for applications that handle large volumes of data in preparation for exascale computing
Barcelona Supercomputing Center-Centro Nacional de Supercomputación (BSC-CNS) researchers have contributed their software and high performance computing expertise to the Adaptive Multi-tier Intelligent Data Manager for Exascale project (ADMIRE). This recently concluded project set out to optimize the management and access to big data, with test cases ranging from weather forecasting and turbulence simulations to planetary scale cover mapping and brain super-resolution imaging.
Its high-quality, open-source tools enable cutting-edge I/O optimization for large-scale HPC infrastructures, supporting diverse scientific disciplines and contributing to the realization of European exascale systems. The project’s advancements align with EuroHPC goals to provide world-class supercomputing and data infrastructure.
BSC researchers played a key role in the success of the ADMIRE project through their work on the development of a software environment that improves the orchestration between the main components and applications of the system. To achieve this, the BSC has amplified and perfectioned three of their already-ongoing software projects: GekkoFS, developed in collaboration with the Johannes-Gutenberg University of Mainz, dataClay and SCORD.
“By integrating these three components with ADMIRE services, we limited the dependency of the applications on the shared storage system, reducing bottlenecks in data input and output (I/O) and ensuring more precise control over system resources. Moreover, we increased the usability of Adhoc Filesystems, like GekkoFS, for the HPC user" explains Ramon Nou, principal investigator of the project from BSC.
The ADMIRE project successfully optimized resource planning. Its analyses allow for a reduction in interferences between applications, resulting in more efficient planning solutions and significant performance improvements. For example, there was an increased application throughput in Deep Learning use cases from 50% to 95%. Similarly, other use cases, such as environmental models like Wacomm++, achieved a 75% performance improvement compared to versions that relied on a Parallel Filesystem.
About the ADMIRE project
ADMIRE (Adaptive multi-tier intelligent data manager for Exascale) is a European-funded project with a budget of €7.9M that started on 1 April 2021 and finished on 30 June 2024, after a 6-month extension. Coordinated by UC3M (Spain), the project brings together a multidisciplinary consortium: BSC (Spain), JGU (Germany), TUDA (Germany), MPG (Germany), FZJ (Germany), DDN (France), Paratools (France), INRIA (France), CINI (Italy), CINECA (Italy), E4 (Italy), PSNC (Poland), and KTH (Sweden).
The ADMIRE project has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement no. 956748. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and Spain, Germany, France, Italy, Poland, and Sweden.