Published by Zhongxin Data | November 2026
The network is no longer a utility in AI data centers — it is the deterministic substrate that decides whether 10,000 GPUs finish a training run in 18 days or 31 days. Spine-Leaf with predictable Clos behaviour is the most deployed topology to make that bet.
If you are provisioning an AI cluster of any meaningful scale in 2026 — a single 64-GPU pod, a 1,024-GPU row, or a multi-rack 16,384-GPU supercomputer — you are almost certainly running a two-tier Spine-Leaf (Clos) fabric, possibly extended to a three-tier super-spine when pods reach tens of thousands of endpoints. This guide is a working engineering reference: it walks through the bandwidth math, oversubscription ratios, ECMP and adaptive routing choices, buffer-depth planning, optics/SerDes co-design, and the failure modes that bite at scale.
We will reference the Mellanox / NVIDIA Quantum and Spectrum switch families Models listed on zhongxindata.hk are available at zhongxindata.hk (Quantum-2 QM9700, Quantum-3 QM3400/QM3500, Spectrum-4 SN5600, Spectrum-3 SN3700/SN4700) — because the design choices below are best understood when tied back to the silicon that actually runs them.
1. Why Clos topology won the AI fabric debate
Three topologies dominate hyperscale and AI fabrics today:
- Spine-Leaf (two-tier Clos) — every leaf connects to every spine; non-blocking between tiers.
- Fat-Tree (k-ary n-tree, often three tiers) — generalization of Clos using identical commodity switches; classic supercomputing choice.
- Dragonfly / Dragonfly+ — group-level all-to-all with long-haul links between groups; minimizes switch count, maximizes optical reach.
For AI training, Spine-Leaf wins on three counts: (a) predictable latency because the maximum hop count is fixed at two switch hops between any two endpoints inside the same pod; (b) graceful incremental scaling — add a leaf, add a spine, repeat; (c) commercial silicon (Spectrum-4, Quantum-2/3) is now purpose-built for it with deep shared packet buffers and adaptive routing features tuned for AI collectives.
2. Building the bandwidth model
The job of the fabric designer is one number: does the worst-case collective fit in the time budget? For an all-reduce across N GPUs using a ring or tree algorithm over RoCE or IB, the per-step time is bounded by the slowest link. Get the spine uplink oversubscription wrong and your 10,000-GPU cluster behaves like a 1,000-GPU cluster with a 10× tail latency problem.
2.1 Oversubscription cheat sheet
| Workload | Recommended Leaf-to-Spine Oversubscription | Why |
|---|---|---|
| Hyperscale AI training (LLM, MoE) | 1:1 (non-blocking) | Collectives saturate east-west. Any oversub shrinks effective bandwidth. |
| HPC simulation (CFD, FEA) | 1:1 to 1.2:1 | Bursty collectives tolerate slight oversub if buffer is deep. |
| Mixed training + inference | 1.5:1 | East-west dominates, but inference east-west is less bursty. |
| General-purpose cloud / multi-tenant | 3:1 to 5:1 | Tenants do not all blast at once; cost wins. |
| Storage (NVMe-oF / Ceph) | 1:1 | Loss = retransmits = tail latency = unhappy storage. |
For AI-only pods the rule is: budget non-blocking, then fight capex. We routinely see 1:1 fabrics where the leaf-to-spine uplinks are QSFP-DD 800G and the spines are QM9700 64-port 400G HDR — that’s 1:1 oversubscription at the port level.
2.2 A worked example — 1,024-GPU pod
Assumptions: 8 GPUs per host (H100/MI300), dual-port ConnectX-7 400G NICs in an LACP-ish active-active mode but effectively using one NIC per rail. Each leaf switch has 32 hosts × 2 NICs = 64 downlinks. 8 leaf switches × 32 hosts = 256 hosts = 2,048 GPUs. That’s only half a pod — double it to 16 leaves for 4,096 GPUs, and you need 16 spine switches if you want 1:1. With Spectrum-3 SN3700 (32× 400G) for leaves and Quantum-2 QM9700 (64× 400G HDR) for spines, you build the classic 16×16 = 256-port non-blocking Clos fabric.
| Tier | Switch | Port count used | Speed | Aggregate |
|---|---|---|---|---|
| Spine | QM9700 (Quantum-2) | 64 × downlinks + 0 uplinks | 400G (NDR) per port | 25.6 Tbps |
| Leaf | SN3700 (Spectrum-3) | 32 × server + 16 × spine | 400G down / 400G up | 12.8 Tbps uplink = 1:1 |
| Server NIC | ConnectX-7 | 2 × PCIe Gen5 x16 | 400G | 400 Gbps per rail |
3. ECMP, adaptive routing, and the dark art of “spraying” collectives
Clos relies on ECMP (Equal-Cost Multi-Path) to spread flows across multiple equal-cost spines. Standard ECMP hashes on the 5-tuple, which works fine for many small flows but penalizes the single elephant flow of an all-reduce: only one spine gets hit.
This is where Adaptive Routing earns its name. NVIDIA Quantum-2 introduced out-of-order spray (packet-by-packet spraying across healthy paths) specifically so that a single NCCL all-reduce uses every spine link simultaneously. Spectrum-4 with SHARP-like behaviour, plus RoCE’s explicit congestion notification (ECN), gets most of the same benefit on Ethernet fabrics.
Design rule of thumb: If your AI job is dominated by > 80% NCCL collectives, adaptive routing + ECN gives you back 25–40% of fabric time. If your job is mostly parameter-server gradient exchanges, standard ECMP is fine.
4. Buffer depth: why it matters more than peak throughput
Marketing materials obsess about Tbps. Real-world Clos fabrics fail because of buffer. During a 16K-GPU all-reduce the incast pattern at any leaf can briefly oversubscribe the destination by 10–50×. Without deep shared buffers, packets drop, TCP/RoCE backs off, and you lose minutes of training time.
| Switch | Shared Buffer | Per-port @ 400G | Incast Tolerance |
|---|---|---|---|
| Spectrum-3 SN3700 | 32 MB shared | ~5 ms at line rate | Good for < 8K GPU |
| Spectrum-4 SN5600 | 160 MB shared | ~25 ms at line rate | Strong to 32K GPU |
| Quantum-2 QM9700 | ~120 MB shared | ~18 ms | Tuned for HPC/AI incast |
| Quantum-3 QM3500 | ~270 MB shared | ~40 ms | Best-in-class incast tolerance |
At scale (8K+ GPUs), Spectrum-4 SN5600 or Quantum-3 are the only sensible choices. Choosing a switch purely on port count without checking buffer depth is one of the most expensive mistakes in AI fabric design.
5. Optics, SerDes, and the 800G → 1.6T transition
The fabric speed roadmap is:
- HDR (200G per lane, 400G port): Mellanox Quantum-2, ConnectX-7. Mature in 2026.
- NDR (400G per lane, 800G port): ConnectX-8 / BlueField-3, Quantum-3. Production at hyperscalers in 2025-2026.
- XDR (800G per lane, 1.6T port): ConnectX-9, Quantum-4 — sampling 2026, volume 2027.
For the switch-to-server segment, the form factor in 2026 is split:
- OSFP for 800G NDR (16×50G PAM4 = 800G; can be split to 2×400G via breakout).
- QSFP-DD112 for 800G in data centers that standardized on QSFP-DD originally.
- OSFP 1600 for 1.6T XDR — new connector, requires pull-tab / heat-sink redesign.
If you are buying greenfield in 2026, specify OSFP-capable switches. If you are extending an existing QSFP-DD estate, stick with QSFP-DD112 modules for one more generation. Zhongxin Data carries both form factors — for example the SN3700 Spectrum-3 32-port 400G switch and the ConnectX-7 400G adapter family are the proven 2026 AI fabric building blocks.
6. When to scale up vs scale out
Two-tier Clos stops scaling practically at ~256 ports per tier, which means ~65,000 endpoints if you push it. For 100,000+ GPU AI data centers (xAI Colossus 100K, Meta multi-MW pods), you need a three-tier super-spine or move to Dragonfly+.
| Scale | Topology | Switch Tier 1 / Tier 2 | Typical silicon |
|---|---|---|---|
| 256–4,096 GPUs | Spine-Leaf (2-tier Clos) | Leaf / Spine | SN3700 + QM9700 |
| 4,096–32,768 GPUs | Spine-Leaf + Pods | Leaf / Spine / Super-Spine | SN5600 + QM9700 / QM3500 |
| 32,768–131,072 GPUs | 3-tier Clos or Dragonfly+ | Leaf / Spine / Super-Spine or Group | SN5600 + QM3500 (Ethernet) or Quantum-3 with long-reach optics |
| > 131,072 GPUs | Dragonfly++ / Custom | Group routing w/ multi-link planes | Custom or QM-class optimized |
7. Operational gotchas at scale
Things that don’t show up in lab test reports but kill production:
- MTU alignment. Set jumbo MTU (typically 9216) end-to-end. One mismatched vSwitch or VPC costs you 30% throughput.
- PFC and ECN tuning. Priority Flow Control on RoCE needs careful queue mapping; default ON is rarely right.
- Subnet Manager pinning on InfiniBand. Don’t let it move with chassis failures; pin to a paired pair of managed SM hosts.
- Cable patching discipline. Label, label, label. A wrong-polarity MPO in a 1,024-port fabric will find you at 3am.
- SNMP / telemetry overload. At 65k ports, naive polling melts your NMS. Use streaming telemetry (gNMI / gNTC) or sFlow cut-through sampling.
9. Where to buy the building blocks
Zhongxin Data supplies the full Mellanox / NVIDIA Spectrum and Quantum switching portfolio, including sealed and bench-tested units, alongside ConnectX-7/8 adapters and OSFP/QSFP-DD optics that match the design rules above.
- Network switches — Spectrum-3 / Spectrum-4 / Quantum-2 / Quantum-3
- InfiniBand & Ethernet adapters — ConnectX-6 / ConnectX-7 / BlueField-2
- Optical transceivers — OSFP / QSFP-DD / QSFP28
- RAID controllers — Broadcom MegaRAID / HBA
Need help designing your AI fabric? WhatsApp +86 13068616914 for sizing, BOM, and bench-tested pricing on multi-rack orders.
Choosing the building blocks: which switch goes where
- Leaf switches sit at every server rack and must match the NIC population on that rack — the leaf tier is where 100G and 200G per port count
- Spine switches aggregate the leaves and should be sized for the worst-case east-west traffic, which is why a 2U 64-port spine often carries more ports than its leaf counterpart
- Match airflow direction to the rack: P2C and C2P chassis are not interchangeable in the same row
Where this topology meets the hardware you can buy today
- The switch families in our line cover the tiers this design assumes: 32-port 100GbE leaf parts, 64-port 2U spine parts and 200GbE QSFP56 options
- InfiniBand fabrics reach the same topology with NDR switches, and the link-budget rules below apply to both
- Ask us for the part number list for your target pod size and we confirm availability per part number
How we verify the switches and NICs we list
- Every unit is cleaned, re-flashed and bench-tested on our own test line before it is listed
- We confirm port count, negotiated link rate, firmware revision and error counters against the part number you send
- Ask for photos of the actual unit and the firmware level before you commit
Related reading
- AI cluster network architecture — the end-to-end architecture this guide assumes
- 100G vs 400G vs 800G — port speed selection for each tier
- QSFP28 vs QSFP-DD vs OSFP — optics for the leaf-to-spine links
- RoCE v2 vs InfiniBand — the transport decision behind the topology
- network switches — the switch models this topology needs
Frequently Asked Questions
What is the difference between spine-leaf and fat-tree for AI clusters?
Fat-tree is the traffic pattern — every node sees the same tiered bandwidth — while spine-leaf (Clos) is one way of building it: many leaf switches cross-connected to a smaller set of spine switches, so any leaf reaches any other leaf over the same number of hops. Clos is the standard implementation because the spine count can stay small while leaf count scales linearly.
How much bandwidth do I need per GPU?
For most large-scale training runs, 400Gb/s per GPU is the working target because the collective communication pattern is latency-bound rather than bandwidth-bound. Below that, all-reduce time starts dominating the training step time. The exact number depends on the model size, the sequence length and how much the code overlaps communication with compute.
Should I use InfiniBand or Ethernet for a spine-leaf fabric?
Both work in this topology. InfiniBand brings native RDMA with a managed fabric and predictable latency; Ethernet with RoCEv2 and lossless configuration is cheaper per port and increasingly capable. If your cluster already has an InfiniBand estate, the fabric management and NIC population are usually the deciding factor rather than the topology itself.
What is the right oversubscription ratio?
For training fabrics, 1:1 is the target because a stalled collective stalls the whole step. Storage and management networks can run at higher ratios such as 4:1 or 8:1 because their traffic is not synchronised. The ratio you can afford also depends on whether your spine tier is sized for the sum of leaf uplinks or for the traffic that actually peaks.
Can I buy the switches for this design from you?
We supply the switch and NIC families this topology assumes, and availability is confirmed per part number. Send the part numbers and quantities for the leaf and spine tiers and we quote per part number with a condition note on the actual units. We do not publish a stock list because committed units move between enquiries.