Most ToR upgrades that go wrong fail before a single cable moves: the team picked a switch generation that does not match their optics plan, forgot that a 100G QSFP28 port breaks out only four ways, or ordered servers whose PCIe slots cannot feed a dual-port 100G adapter. The conclusion up front: a 25G-to-100G ToR refresh is a straightforward project if you plan three things in order — silicon generation first, breakout and oversubscription math second, optics and server NICs third — and if you rehearse the cutover on paper before touching production. This guide walks through each decision in the sequence an infrastructure engineer would actually face it.

What a ToR Upgrade Actually Changes

Top-of-rack switches sit at the edge of your failure domain. When you move a ToR layer from 25G to 100G, every downstream dependency changes with it: the cabling plan, the optics inventory, the server adapters, the leaf-spine uplink bandwidth, and in many cases the power and airflow profile of the rack itself. The bandwidth per server connection goes up 4x (or 8x if you are coming from 10G), but the number of physical ports usually goes down, because a single 100G QSFP28 port replaces four SFP28 lanes. That consolidation is where most of the planning value lives: fewer ports per rack means you need a clear policy for which hosts get dedicated 100G and which ones stay on 25G via breakout.

Treat the upgrade as a redesign of the rack edge, not a like-for-kind swap. Teams that simply replace a 48-port 25G switch with a 32-port 100G switch often discover mid-migration that they cannot fit the old host count at native speed and end up paying for breakout cables they never budgeted.

Baseline Inventory: What to Audit Before You Order Anything

Before comparing switch models, produce a written inventory of what already exists. The minimum list:

  • Current ToR models, firmware versions, and installed optics (type, speed, and vendor of every transceiver).
  • Host-side NIC inventory: which servers have 25G-capable adapters, which are still on 10G, and which have free PCIe slots able to sustain a 100G card.
  • Cabling: length ranges, whether runs are DAC (twinax), AOC, or optical transceivers with fiber, because reach rules change at 100G.
  • Uplink utilization on the existing leaf-spine fabric, measured at peak, not averaged.
  • Power headroom per rack and whether your new switch’s airflow direction matches the cold/hot aisle layout.

This inventory determines your constraints. A shop where 80% of hosts are on 10G and idle will plan a phased rollout; a shop with GPU nodes saturating 25G links will prioritize native 100G for those hosts on day one. The switch shopping comes after, not before.

Switch Silicon Generations: Matching the Platform to Your Roadmap

Mellanox/NVIDIA Spectrum switches are the most common ToR choice for 100G estates, and the generations matter because they define your future upgrade path. Spectrum-1 is a proven, widely deployed 100G platform; Spectrum-2 adds deeper buffers and richer telemetry for larger fabrics; Spectrum-3 carries the estate to 200G per port; Spectrum-4 moves to 800G with OSFP. The table below maps the common models you will encounter to their positions.

Model Silicon Ports Speed per Port Form Factor Typical Role
SN2700 Spectrum-1 32 x QSFP28 100G (breakout 4x25G) 1U Cost-effective ToR leaf for 25/100G estates
SN3800 Spectrum-2 64 x QSFP28 100G 2U High-density 100G leaf when port count outranks speed
SN4600 Spectrum-3 64 x QSFP56 200G 2U 200G-ready spine or dense AI/storage leaf
SN5600 Spectrum-4 64 x OSFP 800G 2U AI cluster fabric and 800G spine layers

The practical rule: if your roadmap stops at 100G for the next several years, a Spectrum-1 platform such as the Mellanox SN2700 100GbE QSFP28 switch is the workhorse choice and the most cost-efficient per port. If you want headroom and denser 100G port counts in the same rack unit budget, the SN3800 64-port 100GbE switch doubles the port count in 2U. If 200G or AI-scale east-west traffic is on the roadmap within 24 months, evaluate Spectrum-3 class hardware now so you do not rebuy optics twice.

Breakout Planning: 4x25G, Oversubscription, and Port Budgets

A 100G QSFP28 port can break out to four 25G SFP28 lanes. That single fact drives most ToR layouts. Work through the math per rack:

  • Count hosts that genuinely need native 100G today: storage heads, hypervisor uplinks, GPU nodes, backup gateways.
  • Break out remaining 100G ports to 4x25G for the general host population, usually with passive breakout DACs for in-rack runs.
  • Reserve uplinks: a typical layout allocates 4-8 ports of the 32 to leaf-spine uplinks, and those uplinks should be native 100G, not breakout.

Oversubscription is the number most teams skip. If 12 hosts at 25G each share one 100G uplink, that is a 3:1 oversubscription ratio toward the spine — acceptable for general workloads, painful for storage replication or backup windows. Write the ratio down per rack before you finalize the port budget, because it tells you how many spine uplinks the new design truly needs.

Optics: DAC, AOC, and Fiber Reach Rules at 100G

At 25G and below, teams lean heavily on passive DACs and rarely think about optics. At 100G the physics tighten: passive twinax reach drops, and the choices multiply. The working rules of thumb:

  • Same-rack and adjacent-rack (roughly under 3 meters): passive DAC remains the cheapest and most reliable option, including breakout DACs for 4x25G.
  • Across a few racks (3-30 meters): AOCs give you lighter cables and better bend tolerance than twinax.
  • Beyond about 30 meters, or between racks with conduit paths: optical transceivers with OM4/MMF or single-mode fiber become the standard.

Two cautions. First, mixing transceiver vendors across a link (transmitter from one brand, receiver from another) is a common source of link-flap tickets; keep each hop consistent. Second, breakout ports sometimes require configuration changes to split the port into 4 lanes; do this deliberately and label both ends, because a mislabeled breakout is the classic root cause of “the link was fine until we recabled.”

Server-Side Readiness: NICs, PCIe Lanes, and Adapter Selection

The switch side is half the project; the host side is where budgets silently double. A server needs a PCIe slot with enough lanes to sustain 100G. As a rule of thumb, a PCIe 3.0 x16 slot can carry a single-port 100G adapter; dual-port 100G cards generally want PCIe 4.0-class slots, or at minimum a x16 PCIe 3.0 slot with acceptance that both ports cannot simultaneously saturate.

For single-port 100G over QSFP28, the Mellanox ConnectX-5 family is the standard fit — for example the ConnectX-5 MCX4621A-ACAT 100GbE adapter — and it pairs cleanly with Spectrum-1 ToR hardware. For dual-port 100G, ConnectX-5 variants such as the dual-port 100GbE ConnectX-5 network card cover top-of-rack hosts that need two independent fabric legs. If you plan toward 200G or InfiniBand-adjacent workloads, the ConnectX-6 generation on PCIe 4.0 is the better landing spot; the dual-port HDR100 ConnectX-6 PCIe 4.0 adapter keeps Ethernet and HDR InfiniBand flexibility on the same card class.

Audit free slots before you buy adapters: a chassis that looks roomy in the spec sheet often has its x16 slots occupied by GPU or HBA cards. This is also the right moment to standardize NIC firmware and driver versions across the fleet, because mixed firmware is the second most common cause of flaky 100G links after optics mismatch.

Cabling, Airflow, and Rack Physical Constraints

100G twinax cables are thicker and stiffer than their 25G counterparts, and cable management failures show up as bent connectors and marginal links rather than clean outages. Plan for it:

  • Verify switch airflow direction matches the rack’s cold/hot aisle; mixing airflow directions in one rack creates hot spots that throttle hardware long before software does.
  • Order cable lengths conservatively; a DAC that is 20 cm too short cannot be “stretched,” and re-ordering mid-migration stalls the whole cut.
  • Label both ends of every breakout cable with the parent port ID, since a 4x25G breakout looks identical to a native 25G DAC at a glance.
  • Check power supply count and circuit capacity on the ToR; dual-powered switches in a rack that assumed single-supply draw can trip breakers during migration.

Migration Sequencing: Cutting Over Without an Outage

The safest migration pattern for a multi-rack estate is parallel-run per rack, not big-bang. The sequence that works in practice:

  1. Stage the new 100G ToR in the same rack, powered but not cabled to production uplinks.
  2. Move or dual-home the highest-value hosts first: storage, virtualization management, GPU nodes. Dual-homing — one leg on the old 25G fabric, one leg on the new 100G fabric — gives you an instant rollback path.
  3. Bring up new fabric uplinks to the spine on dedicated ports, and verify routes and LLDP adjacency before shifting any host traffic.
  4. Migrate general hosts in batches, one rack or one cabinet at a time, keeping a change window with a named rollback owner.
  5. Retire the old ToR only after the last breakout-dependent host has moved, not before; breakout ports are almost always the long tail.

Teams with a maintenance window and no dual-homing option can still do this safely — the difference is that each batch’s rollback plan becomes “recable to the old switch,” so keep the old hardware powered and racked until sign-off.

Validation Checklist Before You Sign Off

Do not accept a migrated rack on “the link is up.” The minimum validation set:

  • Link-level: every port negotiated at the intended speed, FEC (forward error correction) mode consistent per link class, no retrain counters climbing.
  • Throughput: run iperf3 or an equivalent between a sample host and a known-good peer, both directions, and record the number. A 100G link that tops out at 40G usually means a lane or FEC issue on one hop.
  • Latency-sensitive paths: storage and hypervisor heartbeat networks get an RTT baseline comparison before and after.
  • Redundancy: pull one uplink per rack during the window and confirm failover behavior matches the design, not hope.
  • Telemetry: confirm the new switches export counters and events to whatever monitoring you run, because a fabric you cannot see is a fabric you cannot troubleshoot.

Common Failure Modes in First-Generation Upgrades

Four patterns account for most first-attempt pain. One, optics mismatch across a hop — keep transmitter and receiver consistent and buy spares of the same type. Two, forgotten breakout configuration — a port that worked as native 100G in the lab needs an explicit split command to serve 4x25G in production. Three, PCIe slot starvation discovered after adapters arrive — audit chassis interiors first. Four, spine uplink undersizing — hosts get 4x bandwidth while uplinks stay the same, and the fabric becomes congested in ways that look like application problems. Every one of these is caught by the audit and validation steps above, which is the real argument for doing them.

Frequently Asked Questions

Can I mix native 100G ports and 4x25G breakout ports on the same ToR switch?

Yes. Spectrum-based ToR switches are designed for mixed-speed operation, and this is the standard layout: a few native 100G ports for uplinks and high-demand hosts, with breakout DACs serving 25G hosts on the remaining ports. Configure breakout ports explicitly and label both cable ends, since a breakout port appears as a single parent port to the switch OS.

Do I need to replace all server NICs when moving the ToR to 100G?

No, and you usually should not. Keep 25G or 10G hosts on breakout lanes and upgrade adapters only for hosts whose workloads demand native 100G. A ConnectX-5 class adapter provides single- or dual-port 100G for those hosts; the rest of the fleet can migrate to native speed later as part of routine hardware refresh.

How far can a passive DAC run at 100G?

Passive twinax at 100G is reliable at short reach, typically up to around 3 meters depending on cable gauge and platform support. For runs beyond that, move to AOCs (roughly 3 to 30 meters) or optical transceivers with fiber. When reach is uncertain, choose the AOC or optical option; marginal DAC runs produce intermittent link retrains that are expensive to diagnose.

Is Spectrum-1 hardware still a sensible choice for a new 100G ToR layer?

For estates whose roadmap stays at 100G, Spectrum-1 platforms such as the SN2700 remain the most cost-efficient per-port choice and are deployed at scale across the industry. Choose Spectrum-2 or 3 class hardware when you need higher port density, deeper buffering, or a 200G upgrade path within your planning horizon. Match the silicon to the roadmap, not to the newest catalog page.

What oversubscription ratio is acceptable for a 100G ToR design?

It depends on workload. General purpose compute tolerates 3:1 or 4:1 toward the spine without users noticing; storage replication, backup bursts, and east-west AI traffic want 1:1 or close to it. Compute the ratio per rack from measured peak traffic, not vendor defaults, and budget spine uplink ports accordingly — this single number determines whether the fabric feels fast after the upgrade.

If you are scoping a 25G-to-100G refresh and want a second opinion on switch generation, breakout layout, or adapter matching, the fastest path is a short conversation with a parts list attached. Browse current 100G ToR switches, adapters, and server hardware availability on zhongxindata.hk, or message us on WhatsApp at +86 13068616914 with your rack count and host list, and we will help you pressure-test the plan before you spend on optics.

Request a quote for this configuration

Chat on WhatsApp +86 13068616914