Processors Are Fast. Memory Is Not.
A modern CPU core can execute 4-8 instructions per clock cycle. At 5 GHz, that's 20-40 billion operations per second. Main memory (DDR5) takes about 60-80 nanoseconds to respond to a random read request — roughly 300-400 clock cycles at 5 GHz where the processor sits idle, waiting. This gap between processor speed and memory speed is the memory wall, and it's been the dominant performance bottleneck in computing for the last 30 years.
Caches exist to bridge this gap. They're small, fast memories that store copies of frequently accessed data closer to the processor. The cache hierarchy — L1, L2, L3 — trades off size, speed, and cost at each level. Getting the hierarchy right is one of the most important microarchitectural decisions a chip designer makes.
L1 Cache: Speed Above All Else
The L1 cache sits right next to the execution units — physically on the same piece of silicon, sometimes just hundreds of micrometers away. It operates at the processor's full clock speed with a latency of 4-5 cycles. Every modern CPU has separate L1 instruction and L1 data caches (the "Harvard" arrangement), typically 32KB-64KB each.
Why Not Make L1 Bigger?
Because speed and size are directly at odds. A larger SRAM array means longer wire distances, which means higher latency. The L1 cache must respond within a single-digit number of clock cycles to keep the pipeline flowing. Doubling the L1 size might push latency from 4 cycles to 6 cycles, which hurts performance more than the extra cache space helps. This tradeoff has been remarkably stable — L1 sizes have barely changed in 20 years while L2 and L3 have grown enormously.
For a related perspective, see Snapdragon 8 Gen 5 Review: Full Architecture Deep Dive and B.
Apple broke convention with their M1-M4 performance cores, using 192KB L1 instruction caches (3x the industry norm). They could do this because their wider pipeline and reorder buffer can tolerate the slightly higher latency, and the instruction cache benefits particularly from larger size on ARM's variable-length encoding in AArch64.
Associativity
L1 caches are typically 4-way to 12-way set-associative. Higher associativity reduces conflict misses (when two addresses map to the same cache set and evict each other) but increases access latency and power. The L1 data cache also has to handle store forwarding — when a load reads data that was recently written but hasn't reached the cache yet, it needs to grab the value from the store buffer instead.
L2 Cache: The Middle Ground
The L2 cache is private to each core (in most modern designs) and serves as a backstop for L1 misses. Typical sizes range from 256KB (efficiency cores) to 2MB (Apple M4 performance cores, AMD Zen 5). Latency is around 12-15 cycles — fast enough that an L2 hit doesn't stall the pipeline too badly, but slow enough that L1 hit rate matters enormously.
See also: ASML and the EUV Monopoly: Why One Company Controls Advanced.
L2 caches are usually 8-way to 16-way set-associative. Some designs use write-back policies (dirty data stays in L2 until evicted) while others use write-through (writes propagate to L3 immediately). Write-back reduces L3 bandwidth demand but complicates the coherency protocol.
L3 Cache: Shared and Large
The L3 cache (sometimes called LLC — Last-Level Cache) is shared across all cores on the chip. Intel's recent desktop processors have 30-36MB of L3. AMD's Zen 4 desktop chips have 32MB, with the option for 3D V-Cache stacking that adds 64MB more (the Ryzen 7 5800X3D and 7800X3D used this to great effect for gaming).
L3 latency is 30-50 cycles. For workloads with large working sets, the L3 hit rate determines whether the processor spends most of its time computing or waiting for memory. Games, for instance, tend to have large, somewhat unpredictable access patterns that benefit enormously from bigger L3. AMD's 3D V-Cache gaming chips demonstrated 15-25% frame rate improvements in some games from L3 cache alone — no clock speed or IPC improvement needed.
For a related perspective, see 5G and mmWave Chip Design: RF Front-End, Beamforming, and Po.
Slice and Mesh Architecture
In multi-core chips, the L3 is physically distributed across the die. Intel uses a ring bus (older designs) or mesh interconnect (server chips) to connect L3 slices. AMD's Zen architecture distributes L3 across Core Complexes (CCXs), with each CCX having its own L3 slice. Accessing your own CCX's L3 is faster than accessing another CCX's — a NUMA-like effect within a single chip.
Cache Coherency
When multiple cores share data, the caches must stay consistent. If Core 0 writes to address X, Core 1's cached copy of address X must be invalidated or updated. The protocol that manages this — typically MOESI (AMD) or MESIF (Intel) — adds overhead to every shared memory access.
False sharing is a common pitfall: two cores accessing different data that happens to share the same 64-byte cache line. Each write invalidates the line in the other core's cache, causing constant cache line bouncing even though the cores aren't actually sharing data. It's one of the most common performance bugs in multithreaded code, and it's entirely a cache architecture artifact.
The Prefetcher's Role
Hardware prefetchers try to predict future memory accesses and fetch data into cache before it's needed. Modern CPUs have multiple prefetchers working simultaneously: stride prefetchers for sequential access, spatial prefetchers for nearby cache lines, and complex pattern prefetchers that learn from access history. A good prefetcher can effectively reduce memory latency to near-L3 levels for predictable workloads. The catch is that bad predictions waste bandwidth and can evict useful data — getting the prefetcher right is as much art as science.