Common case

Figure 4 shows the one way latency while Figure 5 shows the bandwidth of synchronous remote send operations on the two platforms. The latency is measured using a pingpong benchmark in which two processes on two different machine send synchronous messages to each other back and forth. The bandwidth is measured using a pair of processes on two different machines where one process continuously does synchronous sends to the second process. Because the PCI bus bandwidth(133 MB/s) and myrinet bandwidth(160 MB/s) are very high, the dominant portion of time of small message communication is the processing overhead on the host and the network interfaces. The one-way latency for small messages is 14  s while the peak bandwidth is about 96 MB/s on both platforms.

The VMMC communication model also provides a remote-read capability to grab data from the memory of the remote machine without the involvement of host processor that machine. Figure 6 and 7 show the latency and bandwidth performance respectively. Both measurements are performed using a pair of processes on two different machines where one process repeatedly performs reads from the second process' memory using the remote-read operation.

As expected, we see that the OS has little impact on the data transfer in the common case. Later optimization on NT further reduces the latency by 6  s after our port.

  
Figure 6: Remote fetch latency