The Barcelona Supercomputing Center offers COMPSs to the HPC community, a set of tools that helps developers to efficiently program and execute their applications on distributed computational infrastructures.
This COMPSs release includes new features, such as a new software cache for GPU tasks and support for containers with MPI and MPMD tasks.
The workflow provenance has been extended to support the new Workflow Run Crate profile (v0.1).
COMPSs has been assessed with the EOSC-Synergy SQAaaS platform and obtained the silver badge.
The Workflows and Distributed Computing team at the Barcelona Supercomputing Center-Centro Nacional de Supercomputación (BSC-CNS) is proud to announce a new release, version 3.2 (codename Narcissus), of the programming environment COMPSs.
This version of COMPSs updates the result of the team’s work in the last years on the provision of a set of tools that helps developers to program and execute their applications efficiently on distributed computational infrastructures such as clusters, clouds and container managed platforms. COMPSs is a task-based programming model known for notably improving the performance of large-scale applications by automatically parallelizing their execution.
COMPSs has been available for the last years for the MareNostrum supercomputer and Spanish Supercomputing Network users, and it has been adopted in several recent research projects such as mF2C, CLASS, ExaQUte, ELASTIC, the BioExcel CoE, LANDSUPPORT, the EXPERTISE ETN, in the Edge Twins HPC FET Innovation Launchpad project and in sample use cases of the ChEESE CoE. In these projects, COMPSs has been applied to implement use cases provided by different communities across diverse disciplines as biomedicine, engineering, biodiversity, chemistry, astrophysics, financial, telecommunications, manufacturing and earth sciences. Currently, it is also under extension and adoption in applications in the European projects AI-SPRINT, PerMedCoE, MEEP, CAELESTIS, DT-GEO, ICOS, as well as in the Spanish funded project HP2C-DT. A special mention is the eFlows4HPC project coordinated by the group that aims to develop a workflow software stack where one of the main components is the PyCOMPSs/COMPSs environment.
This version comes with a software cache for GPU tasks. Support for GPU tasks was limited in previous versions to copying the data back and forward from CPU memory to GPU memory for each task. With the current version, the data can be stored in GPU memory and reused in multiple GPU tasks with the subsequent reduction of overheads. The GPU software cache stores the data in GPU memory. If the memory node required for a particular data is full, the runtime will free up objects in that GPU with fewer usages until sufficient space becomes available.
One of the features of COMPSs is that tasks can be multi-threaded or multi-node, therefore having support for MPI tasks. Similarly, support for MPMD MPI tasks (coupling of multiple MPI applications) was added in version 3.0. In addition, support for tasks in containers was added in version 2.8. However, the support for MPI and MPMD tasks in containers required further developments and has been added for this version.
Version 3.0 of COMPSs already included a mechanism to record Data Provenance from the execution of COMPSs applications automatically. This enables reproducibility and replicability of both COMPSs applications and dislib algorithms. The mechanism was extended in version 3.1 to Java COMPSs applications. In version 3.2 it has been enhanced with support for the new Workflow Run Crate profile (v0.1) and improved the structuring of source files for the application between others.
COMPSs 3.2 has been assessed with the EOSC Synergy Software Quality Assurance (SQA) as a Service (SQAaaS) tool[1]. The SQAaaS features an assessment tool that acknowledges the quality attributes of the software, enabling proper recognition through shareable and verifiable digital badges. COMPSs has obtained the silver badge for the following criteria: code accessibility (open and publicly available), code publication under open-source license, availability of code metadata, code documentation availability and formatting, and code publication with semantic versioning.
COMPSs 3.2 comes with other minor new features, extensions and bug fixes.
COMPSs had around 1000 downloads last year and is used by around 20 groups in real applications. COMPSs has recently attracted interest from areas such as engineering, image recognition, genomics and seismology, where specific courses and dissemination actions have been performed.
The packages and the complete list of features are available in the Downloads page. A Docker image is also available to test the functionalities of COMPSs through a step-by-step tutorial that guides the user to develop and execute a set of example applications.
Additionally, a user guide and papers published in relevant conferences and journals are available.
For more information on COMPSs please visit our webpage: http://www.bsc.es/compss
The Workflows and Distributed Computing group at the Barcelona Supercomputing Center aims to offer tools and mechanisms that enable the sharing, selection, and aggregation of a wide variety of geographically distributed computational resources in a transparent way. The research done in this team is based in the former expertise of the group, and extending it towards the aspects of distributed computing that can benefit from this expertise. The team at BSC has a strong focus on programming models and resource management and scheduling in distributed computing environments.
[1] https://sqaaas.eosc-synergy.eu/
