The Barcelona Supercomputing Center offers to the HPC community a set of tools that helps developers to program and execute their applications efficiently on distributed computational infrastructures
This release includes the support for a new parameter type, Concurrent, which enables to exploit further parallelism of the applications; a customizable task failure management mechanism that enables the user to give hints to the runtime about how to manage the application in the case of failed tasks; a new @Compss annotation that supports nesting of COMPSs applications; and with a new support for data collections that enables the detection of data dependencies in a much wider way.
Attend live PyCOMPSs demos at the BSC booth #A-1412
The Workflows and Distributed Computing team at the Barcelona Supercomputing Center is proud to announce a new release, version 2.5 (codename Freesia), of the programming environment COMPSs.
This version of COMPSs updates the result of the team’s work in the last years on the provision of a set of tools that helps developers to program and execute their applications efficiently on distributed computational infrastructures such as clusters, clouds and container managed platforms. COMPSs is a task-based programming model known for notably improving the performance of large-scale applications by automatically parallelizing their execution.
COMPSs has been available for the last years for the MareNostrum supercomputer and Spanish Supercomputing Network users, and it has been adopted in several research projects such as OPTIMIS, VENUS-C, EUBrazil OpenBio, EUBrazil CloudConnect, EUBra-BIGSEA, transplant, MUG, EGI, ASCETIC and TANGO. In these projects, COMPSs has been applied to implement use cases provided by different communities across diverse disciplines as biomedicine, engineering, biodiversity, chemistry, astrophysics and earth sciences. Currently it is also under extension and adoption in applications in the projects NEXTGenIO, mF2C, CLASS, ELASTIC, ExaQUte, LANDSUPPORT, the BioExcel CoE, and the EXPERTISE ETN, as well as in a research contract with FUJITSU.
The new release comes with a new direction type for the tasks' parameters: Concurrent, which enables to further-exploit the parallelism of the applications. A set of Concurrent tasks are enabled to run in parallel by the runtime and the programmer should guarantee exclusive access to the specified concurrent parameter. This exclusive access to the parameter can be guaranteed by a storage backend, for example Hecuba or dataClay, or by using a shared file system. This mechanism will be very useful to implement more efficient reduction schemas, between other uses.
Version 2.5 also comes with a customizable task failure management mechanism that enables the user to give hints to the runtime about how to manage the application in the case of failed tasks. Whereas before this extension a COMPSs task failure would stop the application by raising an error, the extension allows application developers to define the behavior to handle a task error: ignore the failure, cancel all the failed task successors, retry its execution or stop the whole application.
The new release also includes a new @Compss annotation that supports nesting of COMPSs applications. While pure nesting is not supported, COMPSs now supports that a COMPSs task is, in turn, another COMPSs application (that includes new tasks). The nested task can be executed in a subset of the computing infrastructure, assigning to it a node or set of nodes, according to the hints given in the interface. The nesting mechanism supports structured software development and reduces the bottleneck of a single scheduler in the application.
Support for data collections in Python that enables the detection of data dependencies in a much wider way is also present in this new release. This mechanism enables to detect dependencies between elements of a larger collection and the collection itself, that was not supported before. This will enable more efficient codes in the case of algorithms involving collections and also more efficient reduction schemes. This extension is only available for the Python binding, PyCOMPSs.
Other relevant features are: support for multi-node tasks, new functions to synchronize files without opening them, a new master node able to execute tasks in the same process space, partial support for Numba tasks and support for ARM MAP and DDT tools.
COMPSs 2.5 comes with other minor new features, extensions and bug fixes.
At ISC-HPC, PyCOMPSs/COMPSs will have a presence at the BSC booth, with presentations and demos. A live demo on the use of PyCOMPSs with Jupyter notebooks will be delivered at the BSC booth #A-1412 on Tuesday 18th at 2.00 pm.
COMPSs has had around 1000 downloads last year and is used by around 20 groups in real applications. COMPSs has recently attracted interest from areas such as image recognition, genomics and biodiversity, where specific courses and dissemination actions have been performed.
The packages and the complete list of features are available in the Downloads page. A virtual appliance is also available to test the functionalities of COMPSs through a step-by-step tutorial that guides the user to develop and execute a set of example applications.
Additionally, a user guide and papers published in relevant conferences and journals are available.
For more information on COMPSs please visit our webpage: http://www.bsc.es/compss