The next frontier in AI inference infrastructure: Bringing compute near data with high bandwidth compute

What you should know:

• The AI industry's biggest bottleneck is no longer compute - it's moving data.
• Memory-bound workloads are exposing the limits of conventional XPU + HBM architectures.
• Qualcomm High Bandwidth Compute, an innovative near-memory computing architecture, brings compute to the data, dramatically reducing movement overhead.
• The next frontier in AI inference infrastructure will be defined by architectures that deliver more efficiency, performance and scale through optimal placement of memory and compute in close proximity.

For half a century, progress in computing has been told as a story about processors. Yet the workloads now defining the industry - the large generative models behind modern AI - have quietly rewritten the rules. Their performance is no longer governed by how fast a chip can calculate. It is governed by how quickly, and how affordably, data can be delivered to the calculation.

This is the memory wall: the widening gap between the appetite of compute and the ability of memory to feed it. Each new generation of accelerators adds arithmetic horsepower faster than the surrounding memory system can supply operands - and a growing share of every workload's time, energy and cost is consumed not by computing, but by moving data back and forth.

This blog post makes a simple argument: The most consequential gains of the coming decade will not come from making processors faster but instead from a structural change in where computation happens - moving computation closer to the memory itself, so that the most data-hungry operations are performed where the data already lives. We call this idea near-memory computing - and we use the term high bandwidth compute (HBC) for the broad class of technologies that bring compute near memory to break the memory wall.

The problem: A new bottleneck

AI inference on modern AI models is extraordinarily memory intensive. Generating each new token of a response requires reading enormous quantities of model parameters and accumulated context from memory. The arithmetic itself is comparatively cheap; the challenge is supplying the operands fast enough to keep the compute engines busy.

Over successive hardware generations, raw compute capability has climbed steeply. Memory bandwidth - the rate at which data can be delivered to that compute - and storage have climbed far more slowly. In fact, it's estimated that over the past decade the size of transformer models have increased by 240x every two years, while AI hardware memory has only increased by 2x.¹ The result is a structural divergence: an ever-larger fraction of a processor's potential sits idle, waiting for data.

Why scaling memory the conventional way is reaching its limits

The industry's usual answer has been to push more bandwidth through the same basic arrangement: keep the memory and the compute as separate components and widen the pipe between them. That approach is encountering hard physical and economic limits:

• Diminishing returns on interfaces. Widening the connection between memory and processor means ever more physical wires and signaling pins - an approach that grows more costly and complex with every step, with less benefit each time.
• The tyranny of distance. Every bit that travels between separate memory and compute components costs energy and time. At today's scale, that movement - not the computation - dominates the power budget.²
• Premium packaging costs. The most advanced memory integration techniques are expensive and supply-constrained, putting the highest performance out of reach for many deployments.

In short, the conventional high bandwidth memory (HBM) plus XPU architecture asks the data to travel to the computation. When data movement is the bottleneck, that is precisely the wrong direction. Other approaches like embedding massive amounts of SRAM in the XPU also have energy, memory capacity and cost challenges. Stay tuned for future blog posts diving into these topics.

The physics: The hidden cost of moving data

It is tempting to think of memory as a passive store and computation as the active work. But at the scale of modern AI, the act of moving data is itself one of the largest consumers of energy in the entire system. Transporting operands across a chip boundary can cost far more energy than the arithmetic performed on them once they arrive.

This reframes the entire design problem. If most of the cost is in the journey, then the highest-leverage move is to shorten the journey - to perform the most data-intensive operations immediately adjacent to the data, inside the memory package itself, and send only the compact results onward.

The idea: Near-memory computing for a new division of labor

Near-memory computing rests on a deceptively simple premise: place computation directly near the memory and let it handle the operations that are limited by data movement rather than by arithmetic complexity. We refer to this approach and the architectures built on it as high bandwidth compute (HBC). The internal bandwidth of the memory, rather than the narrow external interface, becomes the effective memory bandwidth available to computation. Our innovative implementation of near-memory computing is Qualcomm HBC.

Crucially, this is not about replacing the processor. It is about a smarter division of labor. The main processor, such as a Qualcomm Dragonfly AI Accelerator, continues to do what it does best - complex, flexible, orchestration-heavy work. The HBC takes on the set of operations that it is designed to optimally perform and that are enormously expensive to feed. Each does the part it is suited for. The compute die in HBC can be designed to offer a spectrum of solutions, from fixed-function acceleration to full processor capabilities.

Why this is the right time

Three forces make this approach not just attractive but timely:

• The workloads demand it. Generative AI inference is overwhelmingly memory bound. The operations that dominate are exactly the kind that benefit most from being performed in place.
• The economics favor it. By relieving the need for complex and supply-limited packaging, computation in memory offers a path to high performance at meaningfully better cost and efficiency.
• The ecosystem enables it. Mature, high-volume memory technologies provide a foundation that can be evolved rather than reinvented - a pragmatic route to scale. Just as decisively, the stacking technologies that make near-memory computing practical have themselves matured: advances in 3D integration, and in particular the ability to bond DRAM directly onto logic, are moving from the laboratory into high-volume manufacturing. The building blocks are no longer speculative - they are ready.

The payoff: Scaling on every axis at once

Conventional design forces painful trade-offs: more capacity often comes at the expense of bandwidth; more bandwidth at the expense of cost or power. Near-memory computing loosens these constraints, because the effective bandwidth available to computation is the bandwidth inside the memory - vastly larger than anything that can be exported across an external interface.

The practical consequences compound across the system:

Just as importantly, these gains arrive on a roadmap that scales. Capacity, bandwidth and computational capability can be increased together across successive generations - turning a one-time architectural advantage into a sustained trajectory of improvement. As an example, with Qualcomm HBC Gen 1, Qualcomm Dragonfly AI250 is designed to enable an 18x increase in effective memory bandwidth compared to Qualcomm Dragonfly AI200 with LPDDR5X; Qualcomm Dragonfly AI300 with Qualcomm HBC Gen 2 is designed to enable another stepwise improvement with a 54x increase over Qualcomm Dragonfly AI200.

The outlook: What this means for the industry

If computation continues its migration toward the data, several shifts follow for anyone building or buying AI infrastructure.

Performance metrics are being redefined

Headline compute figures matter less than a system's ability to keep that compute fed. Buyers are increasingly evaluating platforms by delivered performance on real, memory-bound workloads - and by performance per watt and per dollar - rather than by peak theoretical throughput.

Efficiency becomes the competitive battleground

As deployments scale to enormous fleets, energy is both the dominant operating cost and the binding physical constraint. Architectures that minimize data movement will hold a structural advantage that compounds with scale.

Memory becomes an active participant

The long-standing boundary between "where data is stored" and "where data is processed" will blur. Memory will increasingly be understood not as a passive reservoir, but as an active contributor to computation - a shift with implications reaching from chip design to data-center economics.

A paradigm shift

For decades, the processor was the protagonist of computing's story. The defining challenge of the AI era is different. The scarce resource is no longer compute - it is the timely, affordable delivery of data to where it is needed.

The organizations that recognize this shift, and design around it, will set the pace for the next generation of AI. The decisive question is no longer how fast a processor can compute. It is how intelligently a system can limit data movement at all.

Qualcomm HBC with near-memory computing is one of the clearest expressions of that principle: a structural answer to a structural problem. The bottleneck has moved - and so must our thinking.

Stay tuned for our HBC blog post series where we'll dive deeper into HBC vs alternative solutions, performance efficiency, systems architecture, software, manufacturability and much more.

Go Deeper

Does Qualcomm HBC replace existing AI accelerators, or does it work alongside them?

Qualcomm HBC is not a replacement for the processor - it is a smarter division of labor. The main accelerator, such as the Qualcomm Dragonfly AI Accelerator, continues to handle complex, flexible, orchestration-heavy work, while Qualcomm HBC takes on the operations that are memory-bound and enormously expensive to feed through a conventional external interface. Each component does the part it is suited for, and the result is a system that is more capable than either element alone.

Is this a one-generation advantage, or does the near-memory approach scale over time?

Near-memory computing offers something that conventional scaling cannot: the ability to grow capacity, bandwidth and computational capability together across successive generations rather than trading one against another. We designed Qualcomm HBC for sustained trajectory of improvement, not a single-cycle gain - turning an architectural advantage today into a compounding roadmap for the infrastructure decisions ahead. For example, this is already apparent from the estimated effective memory bandwidth improvements with Qualcomm HBC Gen 1 and Qualcomm HBC Gen 2 for Qualcomm Dragonfly products.