NUMA Management

This session, led by Andi Kleen, was titled "NUMA APIs," but talked little about APIs. The real question was how processes should be managed on NUMA systems.

NUMA architectures are, by definition, multiprocessor systems which are organized into nodes of one or more processors. Each node includes a certain amount of local memory. That memory is accessible across the entire NUMA system, but accesses from the local node are much faster than accesses from farther away. The NUMA architecture creates some interesting challenges for operating systems; those which allocate their memory properly will perform better than the others.

There are, according to Andi, three basic allocation techniques which may be used in NUMA systems:

• "Membind" locks process allocations to a specific node. Keeping all process memory on the local node should yield better performance, as long as other references to that memory remain local as well. The membind scheme can lead to allocation failures, however, if the node runs low on local memory.

   

• "Home node" is like membind, except that non-local memory can be allocated if local memory is not available.

   

• "Interleaved" allocations rotate through the nodes, allocating memory from each. In some situations (shared memory regions, perhaps), interleaved allocation schemes can yield better performance.

Different policies work better in different situations; the real question is how those policies should be specified and controlled. One could set up a scheme where each process virtual memory area (VMA) has its own allocation scheme. Linus, however, is opposed to the idea of applications controlling how their own memory is allocated. Rather than having each allocation try to optimize its performance on NUMA systems, this sort of decision is best left to system administrators. There is also, it seems, a shortage of publicly available numbers on just how much performance benefit is to be had from complicated NUMA allocation schemes.

NUMA is an issue that will not go away, but more work must be done before we'll see how Linux will support NUMA in the future.