Executive Summary: Over the last two decades, the field of computational science has seen a dramatic shift towards incorporating high-throughput computation and big-data analysis as fundamental pillars of the scientific discovery process. This has necessitated the development of tools and techniques to deal with the generation, storage and processing of large amounts of data. In this work we present an in-depth look at the workflow engine powering AiiDA, a widely adopted, highly flexible and database-backed informatics infrastructure with an emphasis on data reproducibility.

We detail many of the design choices that were made which were informed by several important goals: the ability to scale from running on individual laptops up to high-performance supercomputers, managing jobs with runtimes spanning from fractions of a second to weeks and scaling up to thousands of jobs concurrently, and all this while maximising robustness. In short, AiiDA aims to be a Swiss army knife for high-throughput computational science. As well as the architecture, we outline important API design choices made to give workflow writers a great deal of liberty whilst guiding them towards writing robust and modular workflows, ultimately enabling them to encode their scientific knowledge to the benefit of the wider scientific community.

Info & Download: https://doi.org/10.1016/j.commatsci.2020.110086.

The first papers is “AiiDA 1.0, a scalable computational infrastructure for automated reproducible workflows and data provenance“, by Talirz, L., Kumbhar, S., Passaro, E. et al., including or PI and WP2 coordinator Nicola Marzari from our EPFL partner.

AiiDA is a “(…) robust open-source high-throughput infrastructure addressing the challenges arising from the needs of automated workflow management and data provenance recording“.

The paper introduces developments and capabilities required to reach sustained performance, with AiiDA supporting throughputs of tens of thousands processes/hour, while automatically preserving and storing the full data provenance in a relational database making it queryable and traversable, thus enabling high-performance data analytics. AiiDA is a constituent part of the INTERSECT IM2D box, an interoperable materials-to-device simulation platform for the development of disruptive electronics.

The second paper is “Materials Cloud, a platform for open computational science“, by Huber, S.P., Zoupanos, S., Uhrin, M. et al., including Nicola Marzari.

Materials Cloud is a “(…) platform designed to enable open and seamless sharing of resources for computational science, driven by applications in materials modelling“.

The paper illustrates the Materials Cloud platform ecosystem, with its LEARN, WORK, DISCOVER, EXPLORE, and ARCHIVE sections, and aiming to support researchers throughout the life cycle of a scientific project and to make their research output FAIR and reproducible. AiiDA also manages the interconnection of our INTERSECT IM2D from/to external materials databases, such as Materials Cloud.

Read more about the papers on https://nccr-marvel.ch/highlights/2020-09SciData.
To discover more our INTERSECT advancements, take a look at the most recent published deliverables (M18+1) on our website.

References:
→ Huber, S.P., Zoupanos, S., Uhrin, M. et al. AiiDA 1.0, a scalable computational infrastructure for automated reproducible workflows and data provenance. Sci Data 7, 300 (2020).
→ Talirz, L., Kumbhar, S., Passaro, E. et al. Materials Cloud, a platform for open computational science. Sci Data 7, 299 (2020).