HBM4 Memory Technology
Breaking the AI Memory Wall

The AI Memory Wall Problem

The "memory wall" is the growing gap between processor compute speed and memory bandwidth. For AI workloads, this gap has become the dominant performance bottleneck. A modern AI accelerator like the NVIDIA H200 can perform over 4 petaFLOPS of FP8 compute, but that compute is useless if the memory system cannot feed data to the processing cores fast enough.

Why AI training is memory-bound

Large language model (LLM) training is fundamentally different from traditional compute workloads. Training a model like GPT-4 or Llama 3 involves:

• Massive parameter storage: A 175B-parameter model requires 350 GB in FP16 just for weights, plus 2-3x more for optimizer states (Adam momentum and variance) and gradients.
• Continuous data movement: Every training step reads all parameters, computes gradients, and writes updated parameters back. At thousands of steps per second, this creates sustained memory bandwidth demand measured in terabytes per second.
• Attention mechanism bottleneck: Transformer attention layers require reading and writing the entire key-value cache for each token, creating memory-bandwidth-proportional scaling.
• Batch processing: Larger batch sizes improve training efficiency but multiply memory bandwidth requirements linearly.

The arithmetic intensity (ratio of compute operations to memory bytes accessed) of transformer training sits at roughly 50-200 ops/byte. Modern AI accelerators deliver 4,000+ TFLOPS but only 3-4 TB/s of memory bandwidth, producing a hardware arithmetic intensity of roughly 1,000-1,300 ops/byte. This mismatch means memory bandwidth, not compute, is the binding constraint for most training workloads.

HBM Evolution: From HBM2E to HBM4

High Bandwidth Memory was first proposed by AMD and SK Hynix in 2011, with the first HBM generation shipping in 2015. Each generation has roughly doubled bandwidth while improving power efficiency and capacity. Understanding this evolution reveals the engineering trajectory that makes HBM4 possible.

The bandwidth trajectory

From HBM2E to HBM4, per-stack bandwidth has increased 4.5x — from 460 GB/s to over 2 TB/s. However, AI model sizes have grown even faster. GPT-3 (2020) had 175B parameters; leading frontier models in 2026 approach 2 trillion parameters. This is why HBM4, despite its massive bandwidth improvement, is still seen as a stopgap rather than a permanent solution to the memory wall.

The transition from HBM3E to HBM4 represents a particularly significant architectural shift. Previous generations increased bandwidth primarily by raising data rates (clock speeds). HBM4 takes a different approach: it doubles the interface width from 1024 to 2048 bits while keeping data rates moderate at 8.0 Gbps. This is a deliberate engineering trade-off — wider interfaces at lower frequencies consume less power per bit and are easier to route through TSVs without signal integrity degradation.

HBM4 Architecture Deep Dive

Through-Silicon Via (TSV) stacking

HBM4 stacks 12 DRAM die (12-Hi configuration, with 16-Hi planned) vertically using through-silicon vias — copper-filled holes etched through each silicon die that create electrical connections between layers. A single HBM4 stack contains over 10,000 TSVs, each approximately 5-6 micrometers in diameter, spaced 40-50 micrometers apart.

The TSV manufacturing process involves four critical steps: via etching (deep reactive-ion etching to create high-aspect-ratio holes), insulation lining (SiO2 deposition to prevent copper diffusion), copper filling (electrochemical deposition), and CMP planarization (chemical-mechanical polishing to create flat surfaces for die-to-die bonding). Each step must achieve sub-micron precision across all 12 die layers — a single misaligned TSV in a stack of 10,000+ renders the entire unit defective.

The logic base die revolution

HBM4 introduces a fundamental architectural change: a dedicated logic base die manufactured on a separate, more advanced process node than the DRAM layers. In previous HBM generations, the base die was essentially a passive redistribution layer (RDL) that routed signals between the DRAM stack and the interposer. HBM4's logic base die is an active silicon chip.

This logic base die enables several capabilities:

• Compute-near-memory (CNM): Simple operations — scatter/gather addressing, data reordering, compression/decompression — can be executed at the base die, reducing round-trip data movement to the host processor.
• Advanced error correction: On-die ECC with full syndrome computation, improving reliability without consuming host-side bandwidth for error handling.
• Power management: Per-channel dynamic voltage and frequency scaling (DVFS), allowing unused channels to enter deep sleep states without affecting active channels.
• Interface flexibility: The logic base die can be customized per customer, allowing different physical interface configurations while reusing the same DRAM stack. NVIDIA, AMD, and Google each specify different base die configurations.

2048-bit interface width

HBM4 doubles the memory interface from 1024 bits to 2048 bits. Each stack has 32 independent channels (up from 16), with each channel carrying 64 bits of data. This wider interface is the primary mechanism for HBM4's bandwidth doubling. The 2048-bit bus requires proportionally more TSVs and interposer traces, which is why advanced 2.5D packaging with silicon interposers or organic bridge technologies is essential.

The wider interface also changes interposer design requirements. A 2048-bit HBM4 interface requires roughly 2,200 signal bumps per stack (including address, command, and power pins), compared to about 1,200 for HBM3E. This increases the minimum interposer area per HBM stack and influences die floorplanning for AI accelerators.

SK Hynix vs Samsung vs Micron: The HBM Manufacturing Race

Three companies control the entire global HBM supply chain. Their competitive dynamics directly determine the pace of AI infrastructure buildout.

SK Hynix: the market leader

SK Hynix established its HBM leadership by securing the sole-supplier position for NVIDIA's A100 (HBM2E) and H100 (HBM3) accelerators. For HBM4, SK Hynix was first to volume production in late 2025, supplying NVIDIA's Blackwell B200/B300 platforms. The company operates dedicated HBM production lines at its Icheon and Cheongju fabs in South Korea, with a new facility in Indiana (US) planned under CHIPS Act incentives.

SK Hynix's key advantage is yield: its HBM3E known-good-die (KGD) yields reportedly exceed 80%, while competitors remain in the 60-70% range. For 12-Hi stacks, compound yield is the critical metric — if each individual die has 95% yield, a 12-die stack yields only 0.95^12 = 54%. SK Hynix's ability to push individual die yields above 97% gives it a substantial cost advantage.

Samsung: closing the gap

Samsung initially fell behind in HBM3 due to thermal and yield issues that delayed its qualification with NVIDIA. By mid-2025, Samsung resolved these problems and secured significant HBM3E orders from both NVIDIA and AMD. For HBM4, Samsung is leveraging its unique position as both a DRAM and logic foundry manufacturer, producing the logic base die in-house on its SF5 (5nm) process — avoiding the cross-company coordination challenges that SK Hynix and Micron face.

Micron: the American contender

Micron entered the HBM market later than its Korean competitors but has been competitive on specifications. Its HBM3E achieved the industry's highest bandwidth-per-watt efficiency. For HBM4, Micron is ramping production at its Hiroshima (Japan) facility, with US-based production planned at its Boise, Idaho headquarters. Micron's 15% market share reflects later market entry rather than technical inferiority — the company's HBM4 products are competitive on bandwidth and power specifications.

Bandwidth and Power Efficiency

HBM4's defining achievement is delivering 2+ TB/s bandwidth per stack while improving power efficiency by approximately 30% over HBM3E. Understanding these specifications requires looking beyond headline numbers.

Bandwidth breakdown

HBM4's 2.05 TB/s per-stack bandwidth is calculated as: 2048 bits (interface width) x 8.0 Gbps (data rate) / 8 bits per byte = 2,048 GB/s. In practice, sustained bandwidth reaches approximately 85-90% of theoretical peak, or 1.74-1.84 TB/s, due to command overhead, refresh cycles, and bank access patterns.

For a GPU with 6 HBM4 stacks (such as the NVIDIA B200), total memory bandwidth reaches 12.3 TB/s theoretical (approximately 10.5 TB/s sustained). This is sufficient to feed 4+ petaFLOPS of FP8 compute at an arithmetic intensity of around 400 ops/byte — well within the range of transformer training workloads.

Power efficiency metrics

HBM4 achieves approximately 2.1 picojoules per bit (pJ/bit), compared to 3.0 pJ/bit for HBM3E — a 30% reduction. For a single stack delivering 2 TB/s, total power consumption is approximately 34 watts. A 6-stack configuration on an AI accelerator consumes roughly 200W for memory alone, out of a total GPU board power of 700-1000W.

The power efficiency improvement comes from three sources:

• Lower signaling voltage: HBM4 operates at 1.0V signaling (down from 1.1V in HBM3E), reducing I/O power quadratically (power scales with V-squared).
• Wider interface at lower frequency: Running 2048 bits at 8.0 Gbps consumes less dynamic power than 1024 bits at 9.2 Gbps for similar bandwidth, because switching power scales linearly with frequency.
• 1-beta DRAM process: The latest DRAM process node reduces cell capacitor leakage and bit-line current, lowering per-bit access energy in the DRAM array itself.

HBM4 in AI Accelerators

HBM4 is not a standalone product — it exists only as a co-packaged component on AI accelerator modules. Every major AI chip platform shipping in 2026 uses HBM4 or is transitioning to it.

NVIDIA Blackwell and HBM4

NVIDIA's Blackwell platform (B200, B300) represents the most aggressive HBM4 deployment. The B300 uses 8 HBM4 stacks in 12-Hi configuration, delivering 384 GB of capacity and approximately 16 TB/s of aggregate bandwidth. NVIDIA co-designs the HBM4 base die with SK Hynix and Samsung, specifying custom PHY (physical layer) interfaces optimized for its NVLink interconnect and proprietary memory controller architecture.

The B200/B300 module uses TSMC's CoWoS-L (Chip-on-Wafer-on-Substrate with local silicon interconnect) packaging to connect the GPU die to the HBM4 stacks. This advanced packaging places HBM stacks within 100 micrometers of the GPU die, minimizing interconnect latency and power. CoWoS-L supports interposer sizes up to 100mm x 100mm — large enough to accommodate a 800mm-squared GPU die plus 8 HBM4 stacks.

AMD MI350 and HBM4

AMD's Instinct MI350 uses 8 HBM4 stacks with a chiplet-based GPU architecture. AMD's approach differs from NVIDIA's monolithic design: the MI350 uses multiple GPU compute dies (GCDs) on a single package, each connected to dedicated HBM4 stacks through a passive silicon interposer. This chiplet architecture allows AMD to use smaller, higher-yielding GPU dies while scaling total compute and memory bandwidth through multi-die integration.

Cost and Supply Constraints

HBM4 is the most expensive memory technology ever mass-produced. Understanding its cost structure explains why AI accelerator pricing remains elevated and why supply constraints persist.

HBM4 cost breakdown

• DRAM die cost: Each HBM4 DRAM die costs approximately $8-12 (on 1-beta process). A 12-Hi stack requires 12 known-good dies: $96-144 in DRAM silicon alone.
• Logic base die: Manufactured on a 5nm-7nm logic process, the base die adds $15-25 per stack.
• TSV processing and stacking: The most expensive step — die thinning (to 30-40 micrometers), TSV formation, micro-bump bonding, and underfill adds $30-45 per stack.
• Testing: Each die is tested before stacking (KGD testing), and the assembled stack undergoes burn-in and speed binning. Testing costs approximately $10-15 per stack.
• Total stack cost: $150-200 in volume, representing a 5-8x price premium per GB over standard GDDR6.

Impact on AI accelerator pricing

HBM4 accounts for 25-35% of total AI accelerator module cost. An NVIDIA B300 module with 384 GB HBM4 (8 stacks x 48 GB) carries approximately $1,200-1,600 in HBM cost alone, contributing to the module's $30,000-40,000 selling price. The HBM cost fraction has increased from approximately 15% (A100 era with HBM2E) to 35% (B300 with HBM4), reflecting both rising HBM complexity and the growing proportion of accelerator value concentrated in memory bandwidth.

What Comes After HBM4?

Even as HBM4 ramps to volume production, the memory industry is already developing next-generation solutions to keep pace with AI's insatiable bandwidth appetite.

HBM4E (High Bandwidth Memory 4 Extended)

Expected in 2027-2028, HBM4E will maintain the 2048-bit interface while increasing data rates to 10-12 Gbps, pushing per-stack bandwidth to 2.5-3.0 TB/s. Stack height may increase to 16-Hi, enabling 64 GB per stack. HBM4E is expected to use the same logic base die architecture as HBM4, easing the transition for accelerator designers.

Beyond HBM: alternative approaches

• Processing-in-memory (PIM): Moving more compute logic into the DRAM array itself, reducing data movement by orders of magnitude. Samsung's HBM-PIM and SK Hynix's AiM (Accelerator-in-Memory) are early implementations, but full PIM architectures require fundamental changes to programming models.
• 3D-stacked SRAM: TSMC and Intel are developing 3D-stacked SRAM caches using hybrid bonding, potentially offering 10-50x bandwidth of HBM at much lower latency. Limited capacity (hundreds of MB to low GB) restricts this to caching, not bulk storage.
• Optical memory interconnects: Replacing electrical TSVs with silicon photonic links between memory and compute dies. This approach could eliminate the bandwidth-distance trade-off but remains in the research phase for memory applications.
• CXL-attached memory pools: Compute Express Link (CXL) 3.0 enables disaggregated memory pools accessed over a high-speed fabric, allowing AI accelerators to access terabytes of external memory beyond their on-package HBM. CXL adds latency (100-300ns vs. 20-50ns for HBM) but vastly expands addressable capacity.

Frequently Asked Questions

Conclusion

HBM4 represents the memory industry's most sophisticated answer to the AI training memory wall. By doubling interface width to 2048 bits, introducing active logic base dies, and improving power efficiency by 30%, HBM4 delivers the 2+ TB/s per-stack bandwidth that modern AI accelerators demand. The technology is a triumph of advanced packaging engineering — stacking 12 DRAM die with 10,000+ TSVs to create a memory component that fits under a postage stamp but delivers bandwidth that entire server racks could not match a decade ago.

Yet HBM4 is not the end of the story. AI model sizes continue growing at 4-5x per year, outpacing even HBM4's bandwidth improvements. The semiconductor industry is already working on HBM4E, processing-in-memory architectures, and photonic interconnects to address the next generation of memory wall challenges. For now, HBM4 is the critical enabler — the component that makes frontier AI training possible, the most supply-constrained part in the semiconductor ecosystem, and a $25+ billion market that shows no signs of slowing down.