AI Training Chip Architectures Compared
The AI accelerator market has exploded, but if you strip away the marketing, there are really only a handful of distinct architectural approaches being used for large-scale model training. I've had the chance to work with or evaluate most of these platforms, and the differences in how they handle matrix math, memory hierarchy, and inter-chip communication tell you a lot more than benchmark numbers on press releases.
NVIDIA: The GPU That Became a Training Machine
Let's start with the obvious one. NVIDIA's H100, based on the Hopper architecture, and its successor the B200 (Blackwell), dominate AI training. The H100 SXM5 delivers 989 TFLOPS of FP16 tensor core performance and 1,979 TFLOPS with sparsity. It has 80GB of HBM3 at 3.35 TB/s bandwidth.
The B200, shipping since late 2025, doubles down: 2 dies connected via a 10 TB/s NV-HBI (NV High Bandwidth Interface) chip-to-chip link, 192GB of HBM3E at 8 TB/s, and roughly 4,500 TFLOPS of FP8 tensor core performance. The dual-die design is honestly NVIDIA admitting that monolithic GPU scaling has hit practical limits — the Blackwell die is fabricated at TSMC N4P, and two ~400mm2 dies connected by a silicon interposer give you the compute density that a single enormous die can't achieve due to yield constraints.
What makes NVIDIA hard to beat isn't just the silicon. It's the software stack. CUDA, cuDNN, NCCL for collective communications, and frameworks like Megatron-LM for distributed training are deeply optimized and battle-tested. When your training run costs $50-100 million in compute, nobody wants to debug custom software.
Google TPU: Purpose-Built and Vertically Integrated
Google's TPU v5p, deployed across their Cloud datacenters, takes a fundamentally different approach. Each TPU v5p chip has two TensorCores, each with a 128x128 systolic array that operates on BF16/INT8 data. The chip-to-chip interconnect is a custom ICI (Inter-Core Interconnect) network that forms a 3D torus topology — each chip connects directly to six neighbors.
The v5p pod scales to 8,960 chips in a single connected unit. That's not a cluster with an InfiniBand fabric — it's a single machine with a direct-connect topology. The bandwidth between adjacent chips is 4.8 Tbps per link. For large model training where all-reduce operations dominate communication time, this topology is incredibly efficient compared to fat-tree InfiniBand networks.
This connects to the ideas in CXL Memory Expansion: Disaggregated Memory Pools and Compute.
Google's TPU v6 (Trillium), announced in 2025, increases per-chip compute by roughly 4x over v5p and maintains the direct-connect ICI fabric. I think the TPU approach — purpose-built silicon with a deterministic communication fabric — is the right architectural direction for training workloads. The problem is you can only use it on Google Cloud.
AMD Instinct: The Viable Alternative
AMD's MI300X, launched in late 2023 and now widely deployed, is their strongest AI training contender. It's a chiplet-based design with eight 5nm XCDs (accelerator compute dies) and four HBM3 stacks, all on a single package using TSMC's CoWoS packaging. Total HBM capacity is 192GB at 5.3 TB/s bandwidth.
The MI300X's key advantage is memory. With 192GB of HBM3, it can hold larger model shards than the H100's 80GB, reducing the degree of model parallelism needed. For LLM inference at scale, this is a significant practical advantage. For training, the picture is muddier — AMD's ROCm software stack has improved enormously, but it's still behind CUDA in terms of library breadth and optimization maturity.
AMD's next-gen MI350 (CDNA 4 architecture, expected late 2026) targets FP8 and FP4 performance aggressively. AMD's claim of 35x inference improvement over MI300X (for specific workloads) is marketing, but a 3-5x real-world improvement seems plausible based on the architectural changes.
Intel Gaudi: The Networking Story
Intel's Gaudi 3 (from the Habana Labs acquisition) is interesting not because of its raw compute — it has 64 TPC (Tensor Processor Core) units delivering roughly 1,835 TFLOPS FP8 — but because of its integrated networking. Each Gaudi 3 chip has 24 x 200Gbps Ethernet ports built into the die. That's 4.8 Tbps of network bandwidth per chip, without needing external NICs or switches for scale-out.
We covered a related topic in DDR5 vs LPDDR5X: Memory Architecture, Bandwidth, and Power E.
For training clusters, networking cost is a massive fraction of total system cost. A 10,000-GPU NVIDIA cluster needs thousands of InfiniBand switches and cables that can cost as much as the GPUs themselves. Gaudi's approach of integrating networking on-chip is, I think, architecturally correct. The execution has been rocky — Intel's software maturity for large-model training lags behind NVIDIA's — but the hardware concept is sound.
Cerebras: The Wafer-Scale Approach
Cerebras is doing something genuinely different. Their CS-3 system uses the WSE-3 (Wafer Scale Engine 3), a single chip that occupies an entire 300mm silicon wafer. It contains 4 trillion transistors, 900,000 AI compute cores, and 44GB of on-chip SRAM spread across the die. No HBM. All the working memory is on-chip, within a few millimeters of the compute.
The advantage is memory bandwidth. When your SRAM is distributed across the die next to the compute cores, you don't have the memory wall problem that plagues GPU architectures. For workloads that are memory-bandwidth limited — and many neural network layers are — this is a legitimate architectural win.
The disadvantage is memory capacity. 44GB of SRAM can't hold a 70 billion parameter model in its weights alone. Cerebras solves this with their "weight streaming" approach, where an external cluster of memory servers feeds weights to the WSE in a pipelined fashion. It works for training, but it adds system complexity and the external memory bandwidth becomes the bottleneck for some workloads.
I honestly think Cerebras has one of the most architecturally interesting approaches, but the market has a strong gravitational pull toward GPU-based systems. Cost per chip isn't publicly disclosed, but estimates put a CS-3 system at $3-5 million.
For a related perspective, see GAA vs FinFET: So Sánh Kiến Trúc Transistor Chi Tiết 2026.
Custom Silicon: Amazon Trainium and Microsoft Maia
The hyperscalers are all building custom training chips. Amazon's Trainium2, available as EC2 p6 instances, uses a 7nm design with high-bandwidth NeuronLink interconnects for scaling. Microsoft's Maia 100, designed with help from AMD engineers, is a custom AI chip for Azure using TSMC 5nm with HBM.
These chips aren't trying to be the fastest single-chip accelerator. They're designed to be cost-effective for the specific workloads each cloud provider runs internally. When you're buying hundreds of thousands of chips, even a 20% cost reduction per training FLOP translates to billions saved.
What Actually Matters for Training Performance
After working with several of these platforms, here's what I've found matters most in practice:
- Memory bandwidth, not peak FLOPS. — Most training workloads are bandwidth-bound for significant portions of the computation. The ratio of FLOPS to memory bandwidth (the arithmetic intensity boundary) determines your actual utilization. The B200's 8 TB/s HBM bandwidth is arguably more important than its 4,500 TFLOPS FP8 peak.
- Interconnect bandwidth at scale. — Training a 400B+ parameter model across thousands of chips means all-reduce communication volume measured in terabytes per iteration. NVLink, Google's ICI, and integrated Ethernet (Gaudi) all attack this problem differently, but they all recognize it as the bottleneck.
- Software maturity. — A chip that's 30% faster on paper but takes 6 months longer to bring up a training run is a net loss. This is why NVIDIA keeps winning — not because their hardware is always the best value, but because CUDA works and engineers know it.
- Power efficiency. — A 10,000-chip training cluster draws 10-20MW. At $0.10/kWh, that's $8-17 million per year just in electricity. Performance per watt is becoming as important as performance per dollar.
The training chip market isn't going to be a single-winner-takes-all outcome, though NVIDIA will maintain the largest share. Google's TPUs are too good for Google to stop building them. AMD's MI series is reaching the maturity threshold where large customers deploy with confidence. And the hyperscalers' custom chips will handle an increasing fraction of internal workloads. That's probably healthy for the industry overall.