Port iota2 to a cloud architecture
The workflow of iota2 (scheduling, pipelining, data access) makes several assumptions like the access to a shared file system containing all input data, the ability to use different ressources (MPI processes, CPU threads, RAM) for the different steps of the processing. These assumptions work well on a single machine (server/workstation) and on a HPC cluster.
Deploying iota2 on a cloud architecture is possible at the VM level: one instance of iota2 runs on a VM in the same way as on a single machine scenario. However, this does not allow to leverage several VMs running in parallel and therefore the ressources available for a iota2 instance are those of a single VM.
It seems that Azure and Amazon are able to simulate HPC clusters on their cloud, but costs are very high. A solution using kubernetes would be more appropriate to being able to deploy iota2 on common cloud infrastructures. The idea would be using several containers in parallel for the different steps of the workflow in the same way as we use MPI processes.
The different containers running the workloads should have access to shared storage volumes, but not all steps of the chain need access to common data and for a particular step the data does not need to be shared between parallel instances.
The fact that iota2 can be started from any of its steps using the configuration file, allows to distribute the workload by updating paths, file names and parameters in the configuration file and reorganize the data storage between steps or even duplicate data on different storages previous to running different steps.
Research on the available technologies (kubernetes mostly, but probably others) needs to be done. Dask-kubernetes (https://kubernetes.dask.org/en/latest/) could be a good solution to simplfy the workflow and minimize code duplication between MPI, PBS and cloud modes.