Computational scientific research involves massive datasets created by today’s cutting-edge instruments and experiments — telescopes, particle accelerators, sensor networks and molecular simulations. Scientific software used to process these massive data sets and extract discoveries from experimental data is typically made up of tens to thousands of smaller functions, blocks of code that handle individual jobs in the long pipeline of data analysis. These programs can be run in their entirety on a single system — be it a laptop, a campus cluster, or a supercomputer — but that uniform approach may not be optimal. Some complex tasks may need to run on high-performance computing resources, and some specialized functions may even be better served by other accelerators or hardware to drive faster scientific discovery.
With a pair of grants to the Universities of Chicago and Illinois from the National Science Foundation totalling $3.14 million, a team led by UChicago CS researchers Ian Foster and Kyle Chard and Daniel S. Katz of the National Center for Supercomputing Applications at the University of Illinois, seek to streamline the process of delegating chunks of data and analysis functions to their ideal destination with funcX, which is a new distributed “function-as-a-service” (FaaS) platform that makes it easier for researchers to easily and automatically delegate their computational workload.
The funcX platform builds upon two existing research technologies: Globus, a research data management platform created at the University of Chicago and Argonne National Laboratory, and Parsl, a Python library for executing parallel workflows created at the University of Chicago, Argonne National Laboratory, and the University of Illinois. funcX will also use Amazon Web Services for hosting management services, and integrate cyberinfrastructure from campuses and national laboratories, such as NCSA, TACC, and XSEDE.
For more on funcX visit : funcx.org