From Scripted HPC-Based NGS Pipelines to Workflows on the Cloud

Abstract

In this paper we describe our initial experiences in the Cloud-e-Genome project with moving the whole exome sequencing pipeline from the scripted HPC-based solution to a workflow enactment system running in the cloud. We discuss shortcomings of the existing approach based on scripts and list benefits that a workflow-based solution can provide. Despite the effort it involved to wrap all required tools in the form of workflow blocks and the restrictions of the dataflow model used to represent workflows we expect the migration to significantly improve the current status of the pipeline. Our target is to enable flexibility, traceability and reproducibility of the solution, so that it can better fit the evolution of tools, data and pipeline itself and allow us to run it at national scale. This work will become foundation for the more complete system that includes variant filtering and interpretation for the diagnostic purposes.

Original document

The different versions of the original document can be found in:

• http://homepages.cs.ncl.ac.uk/paolo.missier/doc/C4Bio.pdf

• http://xplorestaging.ieee.org/ielx7/6844760/6846423/06846521.pdf?arnumber=6846521,

http://dx.doi.org/10.1109/ccgrid.2014.128

• https://dblp.uni-trier.de/db/conf/ccgrid/ccgrid2014.html#CalaXWM14,

https://eprint.ncl.ac.uk/216149,

https://ieeexplore.ieee.org/document/6846521,

https://academic.microsoft.com/#/detail/2025676854