What Is HBM? High Bandwidth Memory for AI GPUs Explained
HBM (high bandwidth memory) is memory made by stacking several memory chips (DRAM) and placing the stack right next to an AI GPU. It lines up more than 1,000 data paths. Each path runs at a moderate speed, but together they move huge amounts of data at once. HBM is hard to make, and only a few companies and factories can make it. That is why it is often in the news during the AI boom.
What you'll learn
- Why HBM is called "stacked memory," and how it is built
- Why HBM sits right next to the GPU (2.5D packaging, CoWoS)
- Getting speed from the number of paths, and how HBM differs from GDDR
- The speed of each generation, and why HBM is hard to make and expensive
- The main makers, and how to read the numbers in the news
Prerequisites: it helps to know that memory (DRAM) is "the place where data in use is kept." "Memory vs Storage: What's the Difference?" explains this.
Where you see it: HBM in the news
"HBM sales sharply raised memory makers' profits." "A new AI GPU will carry the next generation of HBM." Headlines like these are where most people meet the word HBM. HBM is a special memory for GPUs that do AI calculations.
For example, NVIDIA announced its H200 GPU in 2023. It carries 141 GB of HBM3E, one generation of HBM. It can move 4.8 TB of data per second (NVIDIA's specs). A TB (terabyte) is 1,000 GB. Chat AIs and similar services run on GPUs like this, lined up in data centers.
Phones and ordinary PCs, on the other hand, usually do not contain HBM. It is expensive, and they do not need that much speed. Everyday devices use cheaper, easier-to-handle memory in its ordinary form.
Why AI needs to move data fast
AI calculations read a huge number of values, called "weights," from memory. A text AI reads most of its weights each time it produces a word. "Why Are GPUs Used for AI?" explains this in detail.
The amount of data memory can move per second is called "bandwidth." However fast a GPU's calculation units are, they must wait for data if the bandwidth is too low. In AI, bandwidth matters as much as calculation speed.
Memory bandwidth has not kept up with the growth in calculation speed. This problem is called the memory wall. HBM is memory with a much larger bandwidth, made to lower this wall even a little.
The chips inside HBM are DRAM, the same kind as in PC memory. DRAM stores 0s and 1s in tiny containers that hold electric charge. It loses its contents when the power is turned off. What is different is not how HBM stores data, but how its chips are arranged and connected. The rest of this article looks at this in three parts.
How it works 1: stacking DRAM chips
HBM is made of DRAM chips that are ground thin and stacked on top of each other. One such pile is called a "stack." The 2025 standard allows 4 to 16 chips in a stack. Products as of 2026 mostly use 8 or 12.
At the very bottom is a base chip that does not store data. This is called the "base die." It passes data between the DRAM chips stacked above it and the GPU.
The stacked chips are connected by thin metal pillars that run straight through each chip. They are made by drilling deep holes in the silicon and filling them with a metal such as copper. These pillars are called TSVs (through-silicon vias). In 2016, Samsung announced that each of its HBM2 DRAM chips has more than 5,000 TSV holes.
The traditional way to connect stacked chips is to run fine metal wires out from the edge of each chip. TSVs allow many connections across the inside of a chip's surface. They also make the distance between chips very short. This is one reason HBM can line up so many paths.
How it works 2: right next to the GPU
The HBM stacks sit right beside the GPU, inside the same package. A package is the part that encloses chips and connects them to the circuit board outside. "How Are Chips Made?" explains the steps up to packaging a chip.
One stack has 1,024 paths (2,048 in the latest generation, HBM4). Several stacks sit around one GPU. The wiring in an ordinary circuit board is too thick to fit thousands of wires into a small area. So a silicon plate is placed between the chips and the board. Its very fine wiring is built in the same way chips are made. This plate is called an interposer (a go-between plate).
Connecting several chips side by side on one go-between plate like this is called 2.5D packaging. The name means "halfway between laying chips flat (2D) and stacking them (3D)." You can think of HBM as a 3D stack placed beside the GPU in 2.5D.
A leading example of this kind of assembly is CoWoS, a technology from TSMC in Taiwan. TSMC describes it as placing compute chips and HBM side by side on a large silicon interposer. However, not every product with HBM is assembled with CoWoS. Some methods use a different material for the go-between plate, and other companies have their own methods.
How it works 3: more paths, not faster paths
Roughly speaking, the amount memory can move per second is "the number of paths" times "the speed of each path." Each path sends a stream of signals, each one a 0 or a 1. The number of 0s and 1s a path sends per second is given in Gb/s (gigabits per second). 1 Gb/s means 1 billion per second.
HBM standards are set by JEDEC, an industry group that sets standards for semiconductors. HBM3, set in 2022, has 1,024 paths per stack, each running at up to 6.4 Gb/s. Multiplying 1,024 by 6.4 gives about 6,554 gigabits per second. Eight 0s or 1s (8 bits) make one byte, so dividing by 8 gives about 819 GB per second.
Graphics cards for games, on the other hand, often use a type of DRAM called GDDR. GDDR has fewer paths but makes each one very fast. In the GDDR7 standard, set in 2024, one chip has 32 paths. One chip can move up to 192 GB/s, which works out to 48 Gb/s per path.
Each GDDR path is much faster. Even so, HBM has many times more paths, so its total is far larger. HBM can have so many paths precisely because its paths are short.
Running many paths at a moderate speed also saves power. Sending signals very fast over long wires takes a lot of power. HBM's paths are short, reaching only to the chip next door, and their speed is moderate. So moving one bit takes little power. In 2015, AMD said the first HBM moved about 3 times as much data per watt as the GDDR5 of the time. This is AMD's own comparison.
By the numbers: speed by generation
HBM has raised the speed of each path with every generation. In the table below, "per stack" is the amount of data one stack can move per second. The standard values come from JEDEC's announcements.
| Generation | When | Per path | Per stack |
|---|---|---|---|
| HBM | Standard, October 2013 | 1 Gb/s | 128 GB/s |
| HBM2 | Standard, January 2016 | 2 Gb/s | 256 GB/s |
| HBM2E | Standard revised, January 2020 | 3.2 Gb/s | 410 GB/s |
| HBM3 | Standard, January 2022 | 6.4 Gb/s | 819 GB/s |
| HBM3E | Mass production by makers, 2024 | About 9–10 Gb/s | About 1.2–1.3 TB/s |
| HBM4 | Standard, April 2025 | 8 Gb/s | 2 TB/s |
From HBM through HBM3E, one stack has 1,024 paths. HBM4 doubles this to 2,048, and its standard value is 2 TB per second. HBM2E and HBM3E are faster versions of the generation before them, and the makers started using these names. The HBM3E values are a rough guide based on three makers' 2024 announcements. The first-generation values are those of AMD's GPU that first used HBM in 2015 (512 GB/s in total from 4 stacks).
Real products can beat the standard values. HBM4 began shipping in 2026. Per stack, Samsung announced up to 3.3 TB/s, and Micron over 2.8 TB/s. Keep in mind that each maker counts speed and sets test conditions in its own way.
Why HBM is hard to make, and expensive
Thinning chips and stacking them without warping
Up to HBM3E, the height of a stack was limited to 0.72 mm. HBM3E with 12 chips stacked also fits within this height. Divide it by 13 chips, including the base, and each chip gets less than 0.06 mm. For HBM4, the limit was eased slightly, to 0.775 mm. In 2024, SK hynix announced that it made its chips 40% thinner than before for its 12-chip product. Thin chips warp and crack easily. Each maker has its own techniques for stacking them and filling the gaps.
One bad chip ruins the whole stack
The share of good products among all those made is called yield. In a stack, a single bad chip makes the whole stack unusable. For example, stack 8 chips that are each good 99% of the time. The chance that all 8 are good drops to about 92% (0.99 multiplied by itself 8 times). So each chip is carefully tested before it is stacked.
Heat builds up easily
Heat from the chips inside the stack can only escape by passing through the other chips. On top of that, the GPU right next to the stack uses a lot of power. When the temperature rises, charge leaks more easily from DRAM's tiny containers, so the data is harder to keep. Makers also compete on materials and methods that let heat escape more easily.
Wafers and assembly lines run short
Chips are made many at a time on a round silicon plate called a wafer. Micron said in 2024 that HBM3E needs about 3 times as many wafers as PC memory (DDR5) for the same number of bits. The more HBM is made, the fewer wafers are left for other memory. Around the same time, Micron also said its HBM for 2024 was sold out.
Assembly capacity ran short, too. In July 2024, TSMC's CEO said the company had more than doubled its CoWoS production capacity in a year. It planned to double it again the following year. Even so, the outlook was that supply would only catch up with demand in 2025 or 2026.
The main makers, and how to read the news
As of 2026, the main HBM makers are three companies: SK hynix and Samsung of South Korea, and Micron of the United States. All three are large memory makers that design and make their own DRAM.
The first HBM standard was developed mainly by SK hynix and AMD. It became a JEDEC standard in 2013. The first GPUs with HBM were AMD's Radeon R9 Fury series, announced in 2015. Later, GPUs for AI calculations began to use HBM, and demand grew sharply.
For HBM4, SK hynix announced in September 2025 that it was ready for mass production. In 2026, Samsung announced in February that it had started shipping. Micron announced high-volume production in March.
NVIDIA and AMD design GPUs, but they are fabless: they have no factories of their own. They buy HBM from memory makers. The foundry that makes the GPU chip sometimes also handles the assembly. "What Is a Semiconductor Foundry?" explains how the work is split among companies.
The news also often mentions "HBM market share." Each research firm counts share in its own way, and the numbers shift a lot from quarter to quarter. So this article gives no share figures. When you see one, check who counted what, and when.
Common misconceptions
Myth: HBM is a kind of storage for keeping data
In fact, HBM is DRAM: memory where the GPU keeps the data it is working on. Its contents vanish when the power goes off. Its role is completely different from storage such as SSDs, which keep photos and apps. People mix them up because words like "memory," "GB" and "stacking" also come up in news about SSDs.
Myth: Each HBM path is faster than a GDDR path
In fact, it is the other way around: GDDR is faster per path. The 2024 GDDR7 standard allows up to 48 Gb/s per path, while the 2025 HBM4 standard sets 8 Gb/s. HBM comes out ahead in total by having dozens of times more paths.
Myth: Since both are stacked, HBM uses the same technology as the 3D NAND in phone storage
In fact, what gets stacked is different. 3D NAND builds the parts that store data upward in hundreds of layers while a single chip is being made. HBM stacks several DRAM chips that were each finished separately, then links them with TSVs. Also, 3D NAND is for storage that keeps data with the power off, while HBM is working memory.
Learn more
- Compare the types of memory in a table: DDR vs. LPDDR vs. GDDR vs. HBM
- Placing side by side vs. stacking: 2.5D vs. 3D packaging, Silicon interposer vs. organic interposer vs. silicon bridge
- Try the numbers: the memory bandwidth calculator (works out bandwidth from the number of paths and their speed)
- Terms for the next step: microbumps (tiny solder balls that connect chips), hybrid bonding (joining chips without solder), KGD (chips confirmed to be good), chiplets
- See it in figures: a figure of a stack built with TSVs and microbumps is in the list of figures
- Related articles: "Why Are GPUs Used for AI?", and "Beyond Scaling" for new directions such as stacking chips