Functions-as-a-Service (FaaS) are coming to HPC. Already employed in some services-oriented infrastructures, for example, Argonne Scholar Ryan Chard and collaborators are pioneering their introduction into HPC. From use of Singularity and Globus, this is an HPC-centric introduction indeed.
In this video from Singularity User Group, Ryan Chard from Globus at Argonne National Lab discusses "FuncX: A Function Serving Platform for HPC":
"There is a growing need to support the execution of short-duration tasks on High Performance Computing (HPC) infrastructure. To address this need we have developed FuncX: a function serving platform that is designed to provide on-demand execution of stateless functions directly from existing HPC infrastructure. FuncX aims to remove the barriers that exist between functions and large-scale computing infrastructure by simplify execution of function-based workloads, enabling integration of HPC capabilities in applications and analysis pipelines, and ultimately increasing the usability of computing infrastructure. FuncX uses the Parsl scripting library to provide low-latency, scalable, and reliable execution of tasks across heterogeneous computing environments. It securely isolates function executions within containerized Singularity execution environments and leverages Globus, as an identity and access management framework and to automate the movement of functions and data. In this talk I will describe the use cases that have motivated the development of FuncX, present the design of the FuncX execution framework, and present the prototype implementation and early experiences using the system."