Conceptual illustration: a short, wide link to stacked memory next to a long, thin link to distant memory. A faster chip waits when its data arrives slowly.
A faster chip does not make AI faster on its own
A faster AI chip does not make AI faster if the data cannot reach the chip fast enough.
A chip can only work on data it already has. If the data arrives slowly, the chip waits. For much of the time a model spends writing an answer, that wait is the limit, not the calculation.
This article explains the memory that was built to shorten that wait. It is called HBM, short for high bandwidth memory. You will leave knowing what it is, why AI uses so much of it, why there is not enough, and what that means for the cost of AI and of ordinary devices.
What memory does for a processor
A processor does the calculations. Memory holds the data it is working on.
Two numbers describe memory. Capacity is how much it holds. Bandwidth is how fast data can move between the memory and the processor.
Think of a cook and a pantry. Capacity is the size of the pantry. Bandwidth is how many ingredients the cook can carry to the counter each second. A very fast cook with a slow route to the pantry spends most of the day waiting.
What HBM is
HBM is built from the same basic technology as ordinary computer memory, which is called DRAM. What differs is how it is put together. Several memory chips are stacked on top of each other like floors in a building and joined by tiny vertical connections called through-silicon vias. This stack is then placed right beside the processor on a shared base, inside the same package.
That layout gives two benefits. The distance between memory and processor is very short. And the link between them is very wide, which means many bits can move at the same time.
The standard for the latest generation, HBM4, describes a 2,048-bit interface and up to 2 terabytes per second for each stack, according to JEDEC’s April 2025 announcement of the standard. For comparison, a high-end graphics card can use a 512-bit interface to its memory.
Accessible text alternative for this figure
Two layouts compared. In the first, a processor sits in the middle of a shared base with an HBM stack of four memory layers on each side, joined by short, very wide links. HBM4 uses 2,048 bits per stack.
In the second, the processor sits at one end of a circuit board and memory chips sit at the other end, joined by a longer, narrower link. A high-end graphics card can use 512 bits.
The cost of this design is complexity. The stacking is hard to do well, and the shared base, called an interposer, makes the result more expensive than ordinary memory on a circuit board.
A modern AI chip carries several stacks. NVIDIA lists its H100 data-center chip with 80 GB of memory and 3.35 terabytes per second of bandwidth. AMD lists its MI455X with 432 GB of HBM4 across 12 stacks, and a rack of 72 of them with 31 terabytes of HBM4 in total. Those are vendor figures, so read them as what each company claims for its own product.
Why AI needs so much of it
When an AI model writes an answer, it produces one piece of text, called a token, at a time. For each token, the chip has to read the model’s stored numbers, called weights, from memory. It also reads a running record of the conversation so far, which is called the KV cache. We explained the KV cache in Running Open AI Models Yourself.
A research preprint on inference describes this stage as “strictly bandwidth-bound”, meaning memory speed sets the pace. The same paper finds that HBM capacity limits how many users a chip can serve at once, and bandwidth limits how quickly each answer appears.
It gives one example: serving 128 requests with 10,000 reasoning tokens each needs over 1.3 terabytes of memory for the cache alone. This is a preprint, and two of its four authors work at Micron, a memory maker, so read it as informed but not fully independent. The authors note that very large models built from many specialist parts can be limited by routing instead of memory speed. So memory is a major limit, but not the only one.
Here is a simple sum to show the scale. A model with 70 billion weights stored at two bytes each needs about 140 GB just for the weights. That is our own arithmetic, and it already does not fit in the 80 GB of an H100 before any KV cache is counted. Bigger models and longer conversations push the need up further.
We looked at the same pressure from the cost side in why AI inference is changing the way we think about load balancing, and from the efficiency side in Microsoft Maia 300.
Why there is not enough
Four things combine, and they reinforce each other.
- Few makers. HBM supply is concentrated in three companies: SK hynix, Micron and Samsung. In our view, a newcomer would need years to catch up.
- Hard to make. Stacking and joining the chips, and then packaging them next to a processor, takes specialised steps. Micron’s chief operating officer said that more complex future HBM has higher “trade ratios”, which means it uses more factory capacity for each bit, and that this will constrain supply. TSMC’s CEO said on its earnings call that the company has “a big shortage on the back end”, meaning the packaging stage.
- Demand is rising fast. Every AI rack needs a lot of HBM, and the number of racks being built is growing.
- New capacity is slow. Micron’s plans, as summarised from its earnings call, place output from its first new Idaho fab in mid-2027, with later fabs in 2028 and 2030. We did not read these dates in the call itself.
What this month’s results show
Two companies reported results that give a view of both ends of the chain. The figures below come from their own filings unless marked.
| Company | What it reported | What it adds |
|---|---|---|
| TSMC (chip maker, 15 October) | Revenue of NT$1,494.24 billion, up 50.9 percent from a year earlier. Net income of NT$763.06 billion, up 68.7 percent. Its newest 2nm process was 8 percent of wafer revenue. | CEO C.C. Wei said demand is higher than supply and that the gap is “still quite big”. This quote comes from a third-party transcript that has garbled passages. |
| Micron (memory maker, 30 September) | Fourth-quarter revenue of $54.23 billion and full-year revenue of $133.19 billion. It guided to $61.5 billion for the next quarter. | Its chief operating officer said “we really don’t have line of sight to when supply and demand balances.” |
| AMD (chip designer) | Lists 432 GB of HBM4 on each MI455X and 31 TB in a 72-chip rack. AMD expects volume deployments of its Helios rack in the second half of 2026. | Samsung said in March 2026 that it is the preferred HBM4 supplier for the MI455X. This is Samsung’s own statement. |
One caution matters here. The companies saying supply is tight are the ones that sell it, and tight supply helps their prices.
Micron guided a gross margin of about 86 percent for its next quarter, which shows how strong its position is. Treat the forecasts as informed but interested.
Two other views are worth holding alongside them. TrendForce, an independent research firm, says in a 16 October bulletin that HBM stays undersupplied through 2028. That bulletin is a summary of a paid report, so we could not read its numbers.
SK hynix’s own outlook notes that some researchers expect HBM prices to correct after 2026 as capacity grows. Supply being short and prices staying high are not the same claim.
Why this reaches ordinary devices
HBM and ordinary memory draw on the same limited chip-making capacity, according to TrendForce. When makers give more of it to HBM, less is left for the memory in laptops and phones. TrendForce, an independent research firm, forecast on 30 September that conventional DRAM contract prices would rise 10 to 15 percent from one quarter to the next in the last quarter of 2026.
It also reported that higher memory costs are weighing on notebook sales, that some PC brands are cutting SSD sizes in mainstream models, and that smartphone production is being held back. Micron’s COO said the company saw a sequential decline in the memory bits it shipped to mobile and client customers.
TrendForce does not put a number on how much of this is caused by HBM. So the link is real, but its size is not known.
We did not find a confirmed price rise in cloud AI services that comes from HBM. If you buy AI by the hour or by the token, treat memory cost as a risk to plan for and not as a fact today. For the power side of the same story, see why electricity is becoming the real constraint.
How deeply to learn this
Know: Almost everyone who works near AI should know this much. HBM is memory stacked beside the processor, AI reads it constantly, and supply is limited. It explains why AI hardware is costly and why GPU choice is about memory as much as speed.
Use: Engineers who size AI workloads should be able to estimate how much memory a model and its conversations will need, and compare that with the memory and bandwidth of the chips on offer. For the wider picture of chip types, see CPU vs GPU vs TPU vs NPU vs LPU vs DPU.
Master: Hardware and infrastructure architects need the detail. That includes how stacks are built and packaged, how different workloads use memory, and how to plan for supply risk across a multi-year purchase.
What you can ignore for now
You do not need to memorise HBM generation names, speed grades or market share tables. They change every year, and the share figures we found were more than a year old.
Ignore single-source price spikes that circulate on social media, because we could not confirm them. You also do not need to follow every earnings number. The lasting idea is that moving data can limit AI as much as calculating does.
A practical next step
Pick one AI model you use or plan to use and write down three numbers: the size of its weights, how much memory a long conversation adds, and the memory capacity and bandwidth of the chip that runs it. If the first two do not fit comfortably inside the third, memory is your limit, and a faster chip will not fix it.
Go deeper
- Micron fiscal fourth quarter 2026 press release for revenue, guidance and margin.
- TSMC third quarter 2026 Form 6-K for revenue, income and process mix.
- AMD Instinct MI400 series for the MI455X memory figures.
- NVIDIA H100 for its memory capacity and bandwidth.
- Understanding Inference Scaling for LLMs for why inference is limited by memory.
- SK hynix 2026 market outlook for a memory maker’s own view, including the risks it names.
- TrendForce 4Q26 memory price forecast, via I-Connect007 for the effect on ordinary devices.
- JEDEC HBM4 announcement, as republished by Storage Newsletter for the standard’s interface width, speed and stack sizes.
- TrendForce HBM market bulletin, 16 October 2026 for an independent supply outlook through 2028.
- High Bandwidth Memory on Wikipedia for a general reference on how HBM is built.
Figures in this article were checked on 18 and 19 October 2026. Earnings numbers and price forecasts change, so check the sources above before you rely on them.