Socket direct protocol over PCI express interconnect: design, implementation and evaluation

PCI Express (PCIe) has evolved to deliver high throughput, low latency, low power consumption. PCIe has also evolved to support communication across multiple machines. This made PCIe-based interconnects attractive for datacenters and small HPC systems. However, the throughput optimization and compatibility with socket-based applications are the main challenges that need to be addressed to realize the potential of PCIe Interconnect. To address these challenges, we propose using the Socket Direct Protocol (SDP) on top of PCIe as a socket-compatible solution. We designed and implemented SDP as a Linux kernel module. We evaluated our implementation on a PCIe testbed using real-life applications. The results show that SDP achieves up to 98% of the maximum possible bandwidth. We compared the performance of our SDP implementation against SDP over Infiniband and the results show over 10% improvement in file transfer performance and up to 44% reduction of CPU utilization.