Not All Cores Are Created Equal
A GPU looks nothing like a CPU on the inside. Where a CPU dedicates most of its transistor budget to making a few cores very fast at sequential work — branch prediction, out-of-order execution, large caches — a GPU spends those transistors on thousands of simpler cores that collectively tear through parallel workloads. That architectural choice makes GPUs mediocre at running a web browser but extraordinary at matrix multiplication and pixel shading.
NVIDIA's CUDA Core Architecture
Streaming Multiprocessors
NVIDIA organizes its GPUs around Streaming Multiprocessors (SMs). Each SM is a self-contained processing cluster with its own instruction schedulers, register file, execution units, and shared memory/L1 cache. The Ada Lovelace architecture (GeForce RTX 4090) has 128 SMs. The Hopper architecture (H100 data center GPU) has 132 SMs.
Each SM in Ada Lovelace contains 128 CUDA cores (64 FP32 and 64 combined INT32/FP32), meaning the RTX 4090 has 16,384 CUDA cores total. A CUDA core is much simpler than a CPU core — it's essentially a single-precision floating-point ALU with an integer unit, no branch prediction, no out-of-order execution, no complex cache hierarchy of its own.
Warp Execution
NVIDIA GPUs execute threads in groups of 32 called warps. All threads in a warp execute the same instruction simultaneously on different data (SIMT — Single Instruction Multiple Threads). If threads in a warp take different branch paths, the warp serializes — executing each path separately while masking off threads not taking that path. This "warp divergence" kills performance and is the main reason GPU code needs to minimize branching.
Tensor Cores: Purpose-Built for AI
Starting with the Volta architecture (2017), NVIDIA added Tensor Cores — specialized units that perform small matrix multiply-accumulate operations in a single clock cycle. Where a CUDA core does one multiply-add per cycle, a 4th-generation Tensor Core (in Hopper) can perform a 16×16×16 FP16 matrix multiply in one cycle — that's 4,096 multiply-add operations at once.
This connects to the ideas in Chip Design Flow: RTL to GDSII and the Electronic Design Aut.
Tensor Cores support multiple precisions:
- FP64 — for scientific computing (introduced in Ampere)
- TF32 — NVIDIA's 19-bit format that gives FP32-level range with reduced mantissa
- FP16 and BF16 — standard AI training formats
- FP8 (E4M3 and E5M2) — introduced in Hopper for inference
- INT8 and INT4 — for quantized inference
The H100's 528 Tensor Cores deliver about 990 TFLOPS of FP16 throughput with sparsity — roughly 5x the H100's CUDA core FP16 throughput. For transformer-based AI models, Tensor Cores do the heavy lifting while CUDA cores handle everything else.
Ray Tracing Units
RT Cores handle ray-triangle intersection testing and bounding volume hierarchy (BVH) traversal in hardware. Without them, ray tracing on CUDA cores alone would be 5-10x slower. Each RT Core processes ray-BVH traversal steps autonomously, freeing the SMs to do shading work in parallel.
Ada Lovelace's 3rd-gen RT Cores added hardware support for opacity micromap and displaced micro-mesh — compressed geometric representations that accelerate complex scene rendering. The RT Core handles the decompression during BVH traversal, which would be expensive in software.
This connects to the ideas in GAA vs FinFET: So Sánh Kiến Trúc Transistor Chi Tiết 2026.
Honestly, RT Cores matter more for gaming and professional visualization than for AI workloads. Data center GPUs like the H100 include RT Cores but they're rarely used in that context.
AMD's RDNA and CDNA
AMD splits their GPU architectures into two families: RDNA for gaming and CDNA for data center/AI.
RDNA 3 (Radeon RX 7000)
RDNA 3 uses a chiplet design — a 5nm graphics compute die (GCD) paired with six 6nm memory cache dies (MCDs). Each Compute Unit (CU) has 64 stream processors (AMD's equivalent of CUDA cores), dual-issue capable for certain instruction combinations. The RX 7900 XTX has 96 CUs = 6,144 stream processors.
AMD added AI accelerators in RDNA 3 — WMMA (Wave Matrix Multiply Accumulate) instructions that use the existing shader ALUs for matrix operations. It's not dedicated silicon like NVIDIA's Tensor Cores, so raw AI throughput is lower, but it doesn't cost additional die area.
This connects to the ideas in Satellite and Space-Grade Chips: Radiation Hardening and Ext.
CDNA 3 (Instinct MI300X)
CDNA 3 is a completely different architecture optimized for compute. The MI300X uses 3D chiplet stacking with 8 XCDs (Accelerator Complex Dies) on a 5nm process, stacked atop 4 IODs (I/O Dies) on a 6nm process, with 8 stacks of HBM3 memory providing 192GB of capacity and 5.3 TB/s bandwidth. The 304 CUs deliver roughly 1,300 TFLOPS of FP16 throughput.
Memory Architecture Differences
The memory subsystem often matters more than raw FLOPS. NVIDIA's H100 pairs 80GB of HBM3 at 3.35 TB/s bandwidth. The MI300X counters with 192GB of HBM3 at 5.3 TB/s. For large language model inference where the model needs to fit in GPU memory, AMD's larger memory capacity is a genuine advantage — you can serve larger models without model parallelism across multiple GPUs.
Both architectures use L2 caches to reduce HBM bandwidth pressure. H100 has 50MB of L2. MI300X has 256MB of Infinity Cache across its chiplets. This large cache is particularly helpful for workloads with data reuse patterns that fit within it.
The GPU architecture space is evolving fast. The convergence of gaming and AI workloads is driving architectural decisions that would have seemed bizarre a decade ago. Tensor Cores, originally designed for neural networks, now accelerate DLSS frame generation in games. The next generation will likely blur the lines even further.