Designing a 100GbE Data Center Network
100GbE is the standard backbone speed for modern data center fabrics. This guide covers spine-leaf (Clos) topology, NVIDIA Mellanox switch options, and RoCEv2 configuration best practices.
Spine-Leaf Architecture
- Leaf (TOR): Connect to all servers in a rack. Typically 48x25GbE downlinks + 4x100GbE uplinks
- Spine: Interconnect all leaf switches, providing full bisection bandwidth
- East-west traffic: Leaf-spine-leaf (2 hops). North-south: Leaf-spine-border (3 hops)
| Cluster Size | Leaf | Spine | Max Servers |
|---|---|---|---|
| Small (2-4 racks) | 2-4 | 2 | 96-192 |
| Medium (8-16 racks) | 8-16 | 4-8 | 384-768 |
| Large (32-64 racks) | 32-64 | 16-32 | 1536-3072 |
NVIDIA Switch Selection
| Role | Switch | Ports | Rationale |
|---|---|---|---|
| Leaf (TOR) | SN3700 | 32x100GbE | High density, dual-mode |
| Spine | SN3700 | 64x100GbE | Full bisection bandwidth |
| Edge (Small) | SN2100 | 16x100GbE | Cost-effective |
RoCEv2 Configuration Best Practices
- PFC: Enable on RDMA priority (typically priority 3). Disable on others.
- ECN/DCQCN: Enable ECN marking for congestion signaling. Critical at 16+ nodes.
- DSCP: Mark RDMA traffic with class 26 or 106.
- Lossless queue: Reserve 10-20% switch buffer for RDMA lossless queue.
Cabling Guidelines
| Distance | Cable Type | Cost |
|---|---|---|
| <3m (in-rack) | QSFP28 DAC | $30-80 |
| 3-100m | QSFP28 AOC | $200-500 |
| 100-2000m | SR4 transceiver + OM4 | $400-800 |
Frequently Asked Questions
Split 100GbE to 25GbE?
Yes. SN3700 supports QSFP28 breakout: each 100GbE port splits to 4x25GbE. 256x25GbE from one 64-port SN3700.
RoCEv1 vs RoCEv2?
RoCEv1 is L2 only (same subnet). RoCEv2 adds UDP/IP (L3 routable). All modern deployments use RoCEv2.