Building software for quantum-centric supercomputing

IBM has spent years laying the groundwork for a future where quantum and classical high-performance computing (HPC) systems work together to tackle problems neither computational paradigm can solve alone. Now, that future is becoming reality.

Last year, at the Supercomputing 2024 conference in Atlanta, GA, we presented our vision for quantum-centric supercomputing (QCSC), a computational paradigm that we believe will enable the first demonstrations of quantum advantage by the end of 2026. That vision included plans for integrating quantum resources with popular HPC resource management systems like Slurm, as well as proposals for three potential architectures to enable hybrid quantum-classical workflows at scale.

Since then, we've been working with our partners at research organizations like Rensselaer Polytechnic Institute (RPI), STFC Hartree Centre, Cleveland Clinic, Oak Ridge National Laboratory, and the quantum startup Pasqal to bring these plans to life-building open-source tools that real-world HPC users and data center admins can start exploring today.

Explore the quantum plugins for Slurm on GitHub
Explore the Quantum Resource Management Interface on GitHub

These tools have already been put to the test in the real world through our collaboration with the team behind the AiMOS supercomputer at RPI's Future of Computing Institute. Our partnership with RPI has been essential over the past year as we've worked through multiple iterations of our quantum plugins for Slurm and quantum resource management interface.

The RPI team has allowed us to test multiple deployments with an active user base of students and postdocs, helped us explore different strategies for access management and resource allocation, and made invaluable contributions to the arXiv paper describing our efforts. Now, thanks to the integration of AiMOS with the Future of Computing Institute's on-prem IBM Quantum System One, RPI is home to the first QCSC environment ever deployed within the walls of a university.

"This is a major step toward the unification of classical and quantum computing workflows," said Christopher Carothers, Ph.D., chief scientist of RPI's Future of Computing Institute. "It was made possible in part by the collaborative efforts of students, faculty and other members of RPI's research community."

Our work on this project has taught us important lessons, brought us useful insights, and inspired strategic course corrections that are helping to shape the future of quantum-centric supercomputing. Below, we take a quick look at what we've accomplished, and how our vision of quantum-centric supercomputing architectures has evolved since we first shared it last year.

Introducing quantum plugins for Slurm workload manager

Last year, we offered a glimpse at how users might one day leverage workload management systems like Slurm to implement hybrid quantum-classical workflows on a quantum-centric supercomputing architecture. In the months since, we've worked with RPI, STFC Hartree Centre, and Pasqal to realize that vision with a set of open-source quantum plugins for Slurm. Workload managers like Slurm play a vital role in HPC workflows, taking care of resource management and job scheduling, so it's important that we provide plugins that allow them to incorporate quantum resources seamlessly.

We're focusing our initial efforts on Slurm because it is the most popular HPC workload manager in the world, and certainly the most popular among HPC users who also work with IBM quantum computers. For most of the research community, HPC is done through Slurm, and we're adopting that usage pattern to get as many researchers and developers exploring hybrid quantum + HPC workloads as we can. Over time, we hope to support similar initiatives for other resource managers so we can bring quantum to even more of the HPC community.

With the help of our partners, we've built a set of quantum plugins for Slurm using the "Slurm Plug-in Architecture for Node and job Kontrol" or architecture. Integration to resource managers decouples quantum computational resource control and allocation from user code and creates a clear split of responsibilities between HPC data center administrators, HPC users, and HPC application code. Slurm's plugins architecture gives HPC datacenter administrators the ability to add new capabilities to the resource manager without interruption, simply by adding compiled files to the existing environment.

This approach allows us to give full operational control to administrators while maintaining maximum flexibility for users. The result is a familiar user experience for all personas in data centers. Administrators can track and control allocation of resources; researchers and developers use Qiskit in program code and allocate resources during job submission alongside classical CPU, memory and GPUs. To learn more about our quantum plugins for Slurm-including details on the plugins' structure, flow, and general architecture-be sure to read our detailed overview on the project GitHub repo, here.

Developing a vendor-agnostic Quantum Resource Management Interface (QRMI)

HPC users and data center admins can use our open-source quantum plugins to submit jobs incorporating quantum compute resources. However, the plugin itself doesn't directly manage the complexities of controlling the quantum resources, which can vary a great deal depending on hardware vendors and the specifics of individual hardware backends.

That's why, alongside our quantum plugins for Slurm, we've also developed QRMI, the quantum resource management interface. QRMI essentially functions as a thin middleware layer that abstracts away the complexities of controlling the quantum resources of specific hardware backends. Rather than requiring the plugin to manage those complexities directly, QRMI exposes a set of simple APIs that allows the plugin to easily acquire or release hardware, run tasks, and monitor the state of quantum computational resources.

QRMI is written in the high-performance Rust programming language with APIs available in Rust, Python, and C to ensure you can easily integrate it into the computational environment of your choice. These three programming languages give us a foundation that allows us to integrate control over quantum resources to nearly any infrastructure system and framework you can imagine, whether that's resource managers, workflow orchestration tools, the monitoring stack, or something else.

Explore the QRMI project GitHub repository here to learn more about how to install and use our QRMI implementation, and to see examples of the three APIs that are currently available.

Collaborating with RPI to refine our vision for QCSC architectures

In our Supercomputing 2024 blog last fall, we laid out three potential architectures for quantum-centric supercomputing:

• Architecture 1: Separates classical from quantum jobs. In the context of IBM quantum computers, quantum jobs are executions of the Qiskit primitives.
• Architecture 2: A hybrid model where we treat quantum resources like any other resource, and we treat all jobs in the integrated system as hybrid jobs-differentiating and allocating based on the type of resource required by each job.
• Architecture 3: A mixed model that attempts to leverage the benefits of both architecture 1 and architecture 2, while mitigating their drawbacks through more complex configuration and development.

Since then, we've had the opportunity to test these architectures across multiple deployments with the team behind RPI's Future of Computing Institute, which is home to both the AiMOS supercomputer and an IBM Quantum System One. Led by Professor of Computer Science Christopher D. Carothers, the RPI team has been instrumental in helping us determine how to optimize system partitioning and access policies, administer storage and data flow between quantum and classical resources, manage dependencies, and optimize the user experience. These developments will enable quantum-centric supercomputing researchers at RPI and other collaborating institutions to seamlessly transition to an integrated classical-quantum environment that will optimize computational workflows for key applications, such as electronic structure problems, energy-matter interactions, and more.

As our work developing QRMI and the plugins for Slurm in collaboration with RPI has progressed over the past year, we've settled on architecture 2, the hybrid model, as our strategy of choice. One of the benefits of this approach is that once your quantum-classical job starts running, it blocks off all resources required for the job's entire duration. We've already begun to see the benefits of this strategy in research collaborations with STFC Hartree Centre, whose work with IBM under the Hartree National Centre for Digital Innovation Program (HNCDI) highlights the potential for hybrid quantum + HPC workloads powered by our hybrid model QCSC architecture to enable near-term demonstrations of quantum advantage. HNCDI were recently recognized with an IEEE Distinguished Synergy Award for their work driving quantum computing towards adoption.

To learn more about why we settled on the hybrid architecture described above, and to get a technical deep dive on our plugins and QRMI, read our overview paper on arXiv. Please note that, much like the contents of our quantum plugins for Slurm and QRMI GitHub repositories, this paper is very much still a work in progress. This is a collaborative effort involving multiple research organizations, so we invite you to check back in the weeks and months ahead to see how it evolves, and we hope you'll consider joining our collaboration to help shape the future of quantum-centric supercomputing.

Head to the GitHub repos linked above to start exploring the QCSC software tools that are available today and see how you can contribute.