Check out the new USENIX Web site. next up previous
Next: 4.4 Revision #2: Page Migration Up: 4 Implementation Previous: 4.2 Shared Memory Issues

4.3 Revision #1: DLL Aggregation

Due to the many benefits of using dynamically-loaded libraries (e.g., libc), most, if not all processes make use of them, either explicitly or implicitly. Therefore, a substantial number of pages within each process's address space may be shared through the use of DLLs. As discussed above, this sharing inevitably causes pages to be littered across memory, resulting in a drastic size-increase of $ \alpha_i$ for each process $ i$.

The cause of this scattering effect is that we are trying to load library pages into the preferred nodes of processes which initiated read-in from disk, as if these were private pages. To alleviate the scattering effect on the library pages, we need to treat them separately in the NUMA management layer. We implement this simply by ignoring the hint ($ \rho$) that is passed down from the VM layer, and instead, resorting to a sequential first-touch policy, where we try to allocate pages linearly starting with node 0, and fill up each node before moving onto the next node. This ensures that all DLL pages are aggregated together, rather than scattered across a large number of nodes. Table 3 shows a snapshot of the same set of processes under the same workload as in Table 2, but with DLL aggregation employed.

Table 3:  Effect of aggregating pages used by DLLs.

Process $ \rho$ $ \alpha$
syslog 14 0(108) 1(2) 11(13) 14(17)
login 11 0(148) 1(4) 11(98) 15(9)
startx 13 0(217) 1(12) 13(25)
X 12 0(125) 1(417) 9(76) 11(793) 12(928) 13(169) 14(15)
sawfish 10 0(193) 1(281) 10(179) 13(25) 14(11) 15(50)
vim 10,15 0(12) 1(240) 10(5322) 15(4322)
... ... ...

As expected, aggregating DLL pages reduces the number of active nodes per process. However, a new problem is introduced. Due to the extensive use of DLLs, by grouping pages used for libraries onto the earlier nodes, we allocate a large number of pages onto these nodes and quickly fill them. As a result, processes need several of these low address nodes in their active sets to access all of the needed libraries. In the two snapshots shown, this is clearly apparent: after aggregation (Table 3), both nodes 0 and 1 are mapped in all of the process active sets, whereas only node 0 was needed without aggregation (Table 2). With many libraries loaded, we would use up these earlier nodes fairly quickly, and may increase the memory footprint of processes. We explain this in more details in the next section and also describe how to alleviate the extra burden on these earlier nodes.

next up previous
Next: 4.4 Revision #2: Page Migration Up: 4 Implementation Previous: 4.2 Shared Memory Issues