License: Creative Commons Attribution 4.0 International license (CC BY 4.0)
When quoting this document, please refer to the following
DOI: 10.4230/OASIcs.PARMA-DITAM.2021.5
URN: urn:nbn:de:0030-drops-136419
URL: http://dagstuhl.sunsite.rwth-aachen.de/volltexte/2021/13641/
Go to the corresponding OASIcs Volume Portal

Colonnelli, Iacopo ; Cantalupo, Barbara ; Esposito, Roberto ; Pennisi, Matteo ; Spampinato, Concetto ; Aldinucci, Marco

HPC Application Cloudification: The StreamFlow Toolkit (Invited Paper)

pdf-format:
OASIcs-PARMA-DITAM-2021-5.pdf (4 MB)

Abstract

Finding an effective way to improve accessibility to High-Performance Computing facilities, still anchored to SSH-based remote shells and queue-based job submission mechanisms, is an open problem in computer science.
This work advocates a cloudification of HPC applications through a cluster-as-accelerator pattern, where computationally demanding portions of the main execution flow hosted on a Cloud Finding an effective way to improve accessibility to High-Performance Computing facilities, still anchored to SSH-based remote shells and queue-based job submission mechanisms, is an open problem in computer science.
This work advocates a cloudification of HPC applications through a cluster-as-accelerator pattern, where computationally demanding portions of the main execution flow hosted on a Cloud infrastructure can be offloaded to HPC environments to speed them up. We introduce StreamFlow, a novel Workflow Management System that supports such a design pattern and makes it possible to run the steps of a standard workflow model on independent processing elements with no shared storage.
We validated the proposed approach’s effectiveness on the CLAIRE COVID-19 universal pipeline, i.e. a reproducible workflow capable of automating the comparison of (possibly all) state-of-the-art pipelines for the diagnosis of COVID-19 interstitial pneumonia from CT scans images based on Deep Neural Networks (DNNs).

BibTeX - Entry

@InProceedings{colonnelli_et_al:OASIcs.PARMA-DITAM.2021.5,
  author =	{Colonnelli, Iacopo and Cantalupo, Barbara and Esposito, Roberto and Pennisi, Matteo and Spampinato, Concetto and Aldinucci, Marco},
  title =	{{HPC Application Cloudification: The StreamFlow Toolkit}},
  booktitle =	{12th Workshop on Parallel Programming and Run-Time Management Techniques for Many-core Architectures and 10th Workshop on Design Tools and Architectures for Multicore Embedded Computing Platforms (PARMA-DITAM 2021)},
  pages =	{5:1--5:13},
  series =	{Open Access Series in Informatics (OASIcs)},
  ISBN =	{978-3-95977-181-8},
  ISSN =	{2190-6807},
  year =	{2021},
  volume =	{88},
  editor =	{Bispo, Jo\~{a}o and Cherubin, Stefano and Flich, Jos\'{e}},
  publisher =	{Schloss Dagstuhl -- Leibniz-Zentrum f{\"u}r Informatik},
  address =	{Dagstuhl, Germany},
  URL =		{https://drops.dagstuhl.de/opus/volltexte/2021/13641},
  URN =		{urn:nbn:de:0030-drops-136419},
  doi =		{10.4230/OASIcs.PARMA-DITAM.2021.5},
  annote =	{Keywords: cloud computing, distributed computing, high-performance computing, streamflow, workflow management systems}
}

Keywords: cloud computing, distributed computing, high-performance computing, streamflow, workflow management systems
Collection: 12th Workshop on Parallel Programming and Run-Time Management Techniques for Many-core Architectures and 10th Workshop on Design Tools and Architectures for Multicore Embedded Computing Platforms (PARMA-DITAM 2021)
Issue Date: 2021
Date of publication: 02.03.2021

DROPS-Home | Fulltext Search | Imprint | Privacy Published by LZI