Data analysis at the speed of light source experiments
April 14, 2026 | Argonne Leadership Facility News
By Jim Collins
At the upgraded Advanced Photon Source (APS), a powerful X-ray light source in the U.S. Department of Energy’s (DOE) Argonne National Laboratory, new analysis capabilities are changing how experiments unfold. Instead of waiting until an experiment ends, researchers can now use near real-time feedback from the X-ray beamlines to guide their next steps.
APS experiments are now tightly integrated with supercomputers at the Argonne Leadership Computing Facility (ALCF). Building on years of collaboration between APS and ALCF, Argonne researchers created an automated pipeline that streams experimental data from the beamlines to ALCF systems for analysis as it is being collected. The APS and ALCF are DOE Office of Science user facilities.
Linking light and compute
The connection between APS and ALCF is powered by the APS Data Management System and Globus. The APS Data Management System provides a uniform way to connect to data from the approximately 100 unique instruments at the APS. It also keeps track of information about data and experiments at the facility. Globus, a research automation and data management platform developed at Argonne and the University of Chicago, handles the movement of data between the APS and the ALCF’s Polaris supercomputer, automatically running analyses and returning results to the beamline while experiments are still underway.
As instruments like the XPCS beamline at the APS generate massive datasets at microsecond scales, Globus Transfer manages the high-speed movement of this data from the APS to the ALCF, ensuring reliable delivery to the Polaris supercomputer. Globus Flows then orchestrates the entire lifecycle—automatically handling storage, access permissions, and triggering analysis—while Globus Compute executes the specific data-processing functions on the remote supercomputing nodes. This integrated pipeline returns results to researchers in minutes, allowing them to adjust experimental parameters, refine hypotheses, and even synthesize new samples in the wet lab “on the fly” to maximize their allotted beam time.
“The actual data collection is triggering all of the data movement — the storage, the access permissions, the processing on Polaris and the transfer back to the APS,” said Thomas Uram, ALCF data services and workflows team lead. “All of this is happening without any intervention by the scientists.”
Making this seamless access possible involved more than simply connecting APS instruments to remote supercomputers. It also meant changing long-standing policies and operational practices around how DOE computing resources are accessed and used during live experiments.
Traditionally, supercomputing facilities rely on individual user accounts and standard job queues, which can slow down time-sensitive experiments. To overcome these hurdles, the ALCF implemented service accounts — secure credentials tied to a specific experiment rather than an individual researcher — and on-demand queues, which dedicate supercomputer nodes for immediate processing of experimental workloads.
Bringing all the pieces together required extensive collaboration. Teams spanning X-ray science, beamline operations, data management and scientific software worked alongside ALCF staff and Globus developers to map out how each beamline collects data, when to launch processing workflows and how to best integrate APS control systems with Globus and remote supercomputers.
“By combining the expertise of multiple teams with powerful computing resources, we were able to build reliable data processing pipelines that can return analysis results quickly enough to guide experiments as they happen,” said Hannah Parraga, a software engineer at the APS developing scientific data workflows that run on supercomputers for many of the facility’s beamlines.
“The fact that it was so frictionless allowed me to focus on the science I wanted to go after rather than the details of how to manage the data and run the analyses,” Poling-Skutvik said. “That’s the best-case scenario — when the infrastructure exists to let you do what you need to do without limiting your ability to do it.”
The Argonne team’s work is helping to advance DOE’s Genesis Mission, a national AI initiative to build a powerful scientific platform for accelerating discovery science, strengthening national security and driving energy innovation. In particular, the rapid data analysis capabilities have prepared the way for new efforts under the American Science Cloud (AmSC), a cornerstone of the Genesis Mission. AmSC is an integrated, federated platform that connects AI models, curated scientific data, workflows and computing resources across DOE laboratories.
This work was supported in part by the DOE Office of Science’s Advanced Scientific Computing Research (ASCR) and Basic Energy Sciences programs. Access to ALCF computing resources was provided through the DOE ASCR Leadership Computing Challenge award, “Enhancing APS-Enabled Research through Integrated Research Infrastructure,” led by Argonne’s Nicholas Schwarz. Additional funding was provided by DOE’s AmSC project.
The original article can be found here.