Latency Inside the CPU: Microarchitectural Hazards That Steal Microseconds

The Illusion of Constant Execution Time

Two invocations of the same function can execute identical instructions yet take different numbers of CPU cycles. On a path that completes in a few microseconds, a difference of even a few hundred cycles can be significant.

A modern CPU does not simply execute instructions one after another at fixed cost. Much of its design is devoted to avoiding idle time.

When execution reaches a conditional branch, the processor predicts which path will be taken and may begin executing instructions from that path before the condition has been resolved. When one instruction is blocked waiting for data, later instructions that do not depend on it may execute first rather than leaving processing resources idle. Frequently accessed instructions and data are kept in small, fast caches so that the processor does not have to wait for main memory on every access.

The cost of these operations depends on the machine state they encounter. Data may already be in a fast local cache or may have to be fetched from farther down the memory hierarchy. A branch prediction may be correct or may force speculative work to be discarded. Data shared across cores may also take longer to access when the processor has to coordinate between them.

The same source-level code can therefore take a different number of CPU cycles from one invocation to the next. On a path whose total latency is only a few microseconds, delays of tens or hundreds of cycles are no longer negligible.

Pipelines, Dependencies and Branch Prediction

Modern CPUs keep many instructions in flight and may execute ready operations without waiting for all earlier ones to finish. Results are committed in program order, preserving the program's architectural state. Out-of-order execution does not shorten any individual operation's latency but it allows independent operations to proceed during that delay.

One important limit is instruction-level parallelism. In a dependency chain, each operation requires the result of its predecessor:

a = f(a)
a = f(a)
a = f(a)

Execution is serial, causing the latency of each operation to accumulate. The same operations on independent inputs may overlap:

a = f(a)
b = f(b)
c = f(c)

Branches create a different constraint because the processor may not yet know which instruction comes next. Rather than wait for the branch condition to resolve, it predicts the outcome and begins executing along the predicted path. When the prediction is correct, that speculative work is retained. But when it's wrong, the processor discards the speculative work and resumes execution along the correct path.

Frequent branch mispredictions waste speculative work and delay useful execution. Like dependency chains, they reduce the amount of useful work that can overlap and extend the time required to complete the path.

Memory Hierarchy and Locality

When an instruction must wait for data, the delay depends on where that data is found. The processor accesses data through a hierarchy of storage structures with different capacities and latencies. In general, storage closer to a core's execution units provides lower latency but less capacity.

Registers are located within each processor core and hold operands and intermediate results used by instructions.

Caches are managed by hardware and generally allocate and transfer data at cache-line granularity. A load requesting only a few bytes may therefore fetch the entire line containing them (commonly 64 bytes on modern processors).

This behavior rewards two forms of locality:

• Temporal locality: recently accessed data is accessed again.
• Spatial locality: nearby memory locations are accessed close together in time.

Temporal locality increases the likelihood that reused data remains in a nearby cache. Spatial locality allows one cache-line fill to satisfy several loads. Contiguous arrays often benefit from both, whereas scattered accesses and pointer-linked structures are less likely to do so.

When a load cannot be satisfied by a nearby cache, the required cache line must be obtained from a lower cache level, another core, local DRAM, or remote memory. The farther the request must travel, the longer the load generally takes.

The same load may be satisfied from L1 during one invocation and require access to DRAM during another. Code that reuses recently accessed data or accesses nearby addresses is more likely to be served from a cache.

Working Sets, Cache Eviction and Prefetching

The working set of an execution phase is the collection of cache lines it actively uses. If those lines fit in a nearby cache, repeated loads are likely to remain inexpensive. Otherwise, incoming lines may evict them, causing subsequent loads to reach slower levels of the memory hierarchy.

Eviction occurs one cache line at a time. Each cache level uses a processor-specific replacement policy to select which line to remove, typically without maintaining strict least-recently-used ordering.

Hardware prefetchers reduce exposed memory latency by detecting regular access patterns and requesting cache lines before instructions need them. Sequential array traversal produces predictable addresses. In a linked structure, the address of the next object may remain unknown until the current object has been loaded, making it harder to prefetch subsequent nodes. Prefetching helps when a useful cache-line fetch begins early enough to reduce the time a later load must wait.

TLBs and Virtual Memory

Locality matters for address translations, too. Programs access memory through virtual addresses. Virtual memory is divided into fixed-size regions called pages. Address translation maps each virtual page to a region of physical memory while preserving the offset within that page. The translation lookaside buffer (TLB) caches these mappings.

A TLB holds a limited number of mappings. Accessing more distinct pages than it can represent displaces existing entries and later accesses may need to resolve those mappings again. If the translation is absent from the processor's TLB caches, it must be resolved through a page-table walk. This introduces additional memory accesses before the original load or store can complete.

Access patterns spread across many pages place more pressure on the TLB than compact access patterns, even when both read the same amount of data.

NUMA: When RAM Is Remote

Address translation identifies the physical page behind a virtual address. On a NUMA machine, the location of that page determines how far a cache miss must travel.

A NUMA node groups CPU cores with locally attached memory. A core can access memory attached to another node, but the request crosses an interconnect and generally takes longer than an access to local memory.

Consider a thread processing an array:

executing core:                  NUMA node 1
physical pages backing array:    NUMA node 0

If the requested cache line is not already cached, the core in node 1 must obtain it from memory attached to node 0. Nothing in the load instruction or pointer reveals that the access is remote.

Thread and memory placement must therefore agree. Pinning a thread to a core stabilizes where it executes, but does not move its allocated memory. Conversely, migrating a thread to another node can make previously local memory remote.

I/O introduces the same problem. A NIC, its packet buffers, and the thread processing those packets may be placed on different nodes, causing data and cache-line ownership to cross the interconnect.

Cache Coherence, False Sharing and Atomics

When multiple cores access shared memory, latency depends not only on where a cache line is located, but also on which core can modify it. Several cores may hold read-only copies of a line, but a core must acquire writable ownership before modifying it. The coherence mechanism coordinates that transfer and invalidates copies held by other cores.

When different cores repeatedly write to the same line, each may have to reacquire writable ownership from the other. These repeated ownership transfers, known as cache-line bouncing, add cross-core communication latency to the writes.

The cores do not need to modify the same variable. Consider two independent counters:

Core 1 -> counter_a
Core 2 -> counter_b

If both occupy the same cache line,

| counter_a | counter_b | ... |  <- same cache line

each core's write can invalidate the line held by the other. False sharing occurs because coherence operates at cache-line granularity rather than variable granularity, causing logically independent variables to contend for ownership when they occupy the same line.

Atomic operations use the same coherence mechanism as ordinary writes, but a read-modify-write also requires the update to occur indivisibly. Multiple cores can request an atomic update concurrently, but updates to the same value must complete one at a time.

Separating frequently written fields can eliminate false sharing but it cannot eliminate contention over a genuinely shared value, such as an atomic counter updated by several cores.

From Microarchitectural State to Tail Latency

Hardware penalties do not simply add up. Independent memory accesses may overlap, while dependent accesses must wait for preceding results. A sequence of loads that normally hits cache can become substantially slower when several dependent accesses reach DRAM. The instructions are unchanged, but the latency exposed to the program varies.

This distinction matters when interpreting tail latency. Aggregate cache-miss, TLB-miss, or branch-misprediction counts reveal how frequently these events occur, but not how much they delay execution. A few misses on a dependency chain may matter more than many misses whose latency the processor successfully overlaps.

The slowest invocations are not necessarily performing more work. They may be executing the same work under conditions that leave the processor fewer opportunities to hide its cost.