First Principles: building a high performance network in the public cloud

This episode explores the challenges and innovations behind Oracle Cloud Infrastructure (OCI)'s high-performance cluster network, which utilizes RDMA (Remote Direct Memory Access) over Ethernet. Against the backdrop of increasing demand for RDMA support from workloads like Oracle databases and high-performance computing (HPC), the discussion details the complexities of ensuring reliable, lossless packet delivery in a multi-tenant cloud environment. More significantly, the engineers highlight the difficulties of implementing RDMA over Ethernet using the RoCE (RDMA over Converged Ethernet) protocol, focusing on the challenges posed by Layer 2 networks and Priority Flow Control (PFC). The solution involved a Layer 3 network substrate and the use of Explicit Congestion Notification (ECN) and DCQCN (Data Center Quantized Congestion Notification) for congestion control, providing tailored quality of service for different workloads such as HPC, Autonomous Database, and GPU-based machine learning. For instance, HPC workloads prioritize ultra-low latency, while GPU workloads emphasize high throughput. The team also discusses network optimizations like flow shifting to improve performance and the provision of topology information to schedulers for better resource allocation. This innovative approach to RDMA networking in OCI delivers sub-microsecond latencies, even at massive scale, setting it apart from other cloud providers and showcasing a commitment to performance optimization for demanding customer workloads.