Jupiter evolving
Leon Poutievski, Omid Mashayekhi, Joon Suan Ong et al. · ACM SIGCOMM Conference (SIGCOMM) · 2022
We present a decade of evolution and production experience with Jupiter datacenter network fabrics. In this period Jupiter has delivered 5x higher speed and capacity, 30% reduction in capex, 41% reduction in power, incremental deployment and technology refresh all while serving live production traffic. A key enabler for these improvements is evolving Jupiter from a Clos to a direct-connect topology among the machine aggregation blocks. Critical architectural changes for this include: A datacenter interconnection layer employing Micro-Electro-Mechanical Systems (MEMS) based Optical Circuit Switches (OCSes) to enable dynamic topology reconfiguration, centralized Software-Defined Networking (SDN) control for traffic engineering, and automated network operations for incremental capacity delivery and topology engineering. We show that the combination of traffic and topology engineering on direct-connect fabrics achieves similar throughput as Clos fabrics for our production traffic patterns. We also optimize for path lengths: 60% of the traffic takes direct path from source to destination aggregation blocks, while the remaining transits one additional block, achieving an average block-level path length of 1.4 in our fleet today. OCS also achieves 3x faster fabric reconfiguration compared to pre-evolution Clos fabrics that used a patch panel based interconnect.
Cited by 233 works. doi:10.1145/3544216.3544265
Authors
- Leon Poutievski
- Omid Mashayekhi
- Joon Suan Ong
- Arjun Singh
- Mukarram Tariq
- Rui Wang
- Jianan Zhang
- Virginia Beauregard
- P. Conner
- Rishi Kapoor
- Stephen Kratzer
- Hong Liu
- Karthik Nagaraj
- Samir Sawhney
- Ryohei Urata
- Lorenzo Vicisano
- Kevin Y. Yasumura
References
- OpenFlow
- A scalable, commodity data center network architecture
- VL2
- Principles and Practices of Interconnection Networks
- c-Through
- Making intra-domain routing robust to changing and uncertain traffic demands
- Jupiter Rising
- Technology-Driven, Highly-Scalable Dragonfly Topology
- OSA: An Optical Switching Architecture for Data Center Networks With Unprecedented Flexibility
- 1296-port MEMS transparent optical crossconnect with 2.07 petabit/s switch capacity