Designing a 100GbE Data Center Network
100GbE is the standard backbone speed for modern data center fabrics. This guide covers spine-leaf (Clos) topology, NVIDIA Mellanox switch options, and RoCEv2 configuration best practices.
Spine-Leaf Architecture
- Leaf (TOR): Connect to all servers in a rack. Typically 48x25GbE downlinks + 4x100GbE uplinks
- Spine: Interconnect all leaf switches, providing full bisection bandwidth
- East-west traffic: Leaf-spine-leaf (2 hops). North-south: Leaf-spine-border (3 hops)
| Cluster Size | Leaf | Spine | Max Servers |
|---|---|---|---|
| Small (2-4 racks) | 2-4 | 2 | 96-192 |
| Medium (8-16 racks) | 8-16 | 4-8 | 384-768 |
| Large (32-64 racks) | 32-64 | 16-32 | 1536-3072 |
NVIDIA Switch Selection
| Role | Switch | Ports | Rationale |
|---|---|---|---|
| Leaf (TOR) | SN3700 | 32x100GbE | High density, dual-mode |
| Spine | SN3700 | 64x100GbE | Full bisection bandwidth |
| Edge (Small) | SN2100 | 16x100GbE | Cost-effective |
RoCEv2 Configuration Best Practices
- PFC: Enable on RDMA priority (typically priority 3). Disable on others.
- ECN/DCQCN: Enable ECN marking for congestion signaling. Critical at 16+ nodes.
- DSCP: Mark RDMA traffic with class 26 or 106.
- Lossless queue: Reserve 10-20% switch buffer for RDMA lossless queue.
Cabling Guidelines
| Distance | Cable Type | Cost |
|---|---|---|
| <3m (in-rack) | QSFP28 DAC | Quoted |
| 3-100m | QSFP28 AOC | Quoted |
| 100-2000m | SR4 transceiver + OM4 | Quoted |
Designing the topology: when two tiers are enough
- Two tiers — leaf and spine — carry most data center designs and keep the failure domain small
- Add a third tier only when the spine layer becomes the constraint, not by default
- Plan for a topology you can operate: more tiers means more cabling and more configuration surfaces
Sizing leaf and spine bandwidth with worked numbers
- Start from server ports, then size the uplinks so the spine tier never becomes the ceiling
- A common rule is one uplink per leaf server port group, balanced across spine units for resilience
- The 100GbE port count on a 32-port or 64-port leaf switch sets how many servers you can attach per switch
Optics, cabling and the distances that force decisions
- DAC and AEC cables are the norm inside a rack and across an adjacent row; optics take over beyond about 5 metres
- Front-panel density is a real constraint — a 64-port switch occupies 2U and a full rack width
- Budget for spare ports at the top of each switch so a failure can be absorbed without a re-cable
Related reading
- Spine-leaf architecture — the topology this guide builds on
- Data center cabling guide — cabling and optics detail
- QSFP28 vs QSFP-DD vs OSFP — connector selection
- Network switches — category hub
Frequently Asked Questions
How do I design a 100GbE data center network?
Start from the number of server ports you need, group them onto leaf switches, then give every leaf enough uplink bandwidth to the spine tier so the core is never the constraint. Add cabling and optics to the plan once the topology is fixed, because they follow from distance and density. If you tell us your server count and port target, we can size the switch bill of materials with you.
How many servers can one 100GbE leaf switch serve?
A 32-port switch serving 25GbE or 50GbE per server doubles the server count; at 100GbE per server the port count is the limit. On current 64-port platforms you get 64 server ports per switch. Leave headroom at the top of each switch so a failed link can be absorbed without a re-cable.
Do I need 100GbE for a small data center or a lab?
For storage-heavy or east-west-heavy traffic, yes, because 25GbE becomes the bottleneck before the servers do. For a small lab or a homelab-class build, 25GbE adapters with a 100GbE uplink to a single switch can give you most of the benefit at a fraction of the cost, and you can step up per port later.
What uplink ratio should a leaf switch have to a spine?
Match the uplink bandwidth to the aggregate downlink demand and keep it balanced across spine units so a spine failure does not take out more than its share. A common starting point is one high-speed uplink per leaf with the spine tier sized so no single link is a bottleneck during a rebuild or rebalance.
DAC, AOC or optics for 100GbE?
DAC is the cheapest and lowest-latency option for in-rack and adjacent-row runs, typically to about 3 to 5 metres depending on the cable. AOC handles longer in-rack spans with the transceiver integrated. Optical transceivers take over beyond that, and the choice also depends on the switch’s breakout support and the optic modes you standardise on.
How do you verify a used 100GbE switch before shipping?
We clean, re-flash and bench-test on our own test line, then confirm port count, port speed, airflow direction and firmware level against the part number, and check the fan configuration. We will tell you whether a unit is an ONIE build or carries a network OS. Ask for photos of the actual unit before you commit.