For nearly two decades the data center fabric debate has been framed as InfiniBand vs Ethernet. In 2026 the question is sharper than ever. Large language models with hundreds of billions of parameters, distributed training jobs that must keep thousands of GPUs in lock-step, and inference fleets pushing tokens at millisecond budgets — each one of these workloads pressures the network in ways that a generic 10 GbE closet switch never had to face. The wrong choice of fabric can quietly double your training time, push power and cooling budgets past their limits, and lock you into a vendor ecosystem that you cannot easily escape.

At Zhongxin Data we supply both sides of that ecosystem: NVIDIA Mellanox InfiniBand adapters and Quantum/Spectrum switches, and a deep catalog of Broadcom-, Intel-, and Marvell-based Ethernet gear. This guide is the comparison we walk our customers through before they commit a single dollar to a new cluster. It is vendor-neutral on the protocol question, and very specific on the numbers.

Quick Answer: When to Pick Which

  • Pick InfiniBand if you run tightly coupled HPC or AI training, need the lowest possible tail latency at scale, and want NVIDIA’s vertically tuned stack (GPUDirect RDMA, SHARP, NCCL).
  • Pick Ethernet if you run a mixed cloud, storage, and AI workload, want multi-vendor optics and switches, or are deploying 200G/400G/800G fabrics on a converged leaf-spine.
  • Pick Spectrum-X or Ultra Ethernet if you want Ethernet’s openness but also need the lossless, low-latency behavior that AI fabrics demand.

The Underlying Protocols Are Not the Same Animal

InfiniBand is defined by the InfiniBand Trade Association (IBTA). It is a single, end-to-end specification that covers the physical layer, the link layer, the transport layer, the software verbs interface, and the management agents. Every compliant device — HCA, switch, cable, transceiver — has to pass IBTA’s interoperability logo tests. There is no “InfiniBand over someone else’s L2”; it is the L2.

Ethernet, in contrast, is just IEEE 802.3. It defines a physical and MAC layer. Everything that AI fabrics care about — lossless behavior, congestion signaling, RDMA semantics, multi-pathing — is bolted on top by other standards bodies and industry consortia:

  • RDMA: RoCE v2 (RDMA over Converged Ethernet, RFC 8852) or iWARP.
  • Congestion control: DCQCN, PFC, ECN, and now the new Ultra Ethernet (UE) fabric spec.
  • High availability: EVPN-VXLAN, MLAG, MC-LAG.
  • Storage: NVMe-oF, iSCSI, SRP.

That split is both Ethernet’s strength (you can mix and match) and its weakness (the parts do not always interop as cleanly as IBTA’s tested stacks).

Side-by-Side: The Numbers That Matter

Attribute InfiniBand (NDR / XDR) Ethernet (RoCE v2 / Spectrum-X)
Standard link speeds NDR 400 Gb/s, XDR 800 Gb/s (8x of 100G/200G PAM4) 100G, 200G, 400G, 800G (IEEE 802.3ck, 802.3df)
Switch port latency (typical) 90–150 ns (cut-through, IBTA spec) 300–500 ns (store-and-forward or cut-through)
End-to-end GPU-to-GPU (typical 8-hop) ~1.5–2.5 µs ~3–6 µs (RoCE v2), ~2.5–4 µs (Spectrum-X with adaptive routing)
Native RDMA Yes, mandatory in the spec Via RoCE v2, requires lossless L2 (PFC + ECN)
Congestion control Hardware, deterministic, per-packet Software-assisted (DCQCN) or vendor-tuned (Spectrum-X)
Adaptive routing Built into Quantum-2/Quantum-X switches Optional in UE, optional per-vendor
SHARP / in-network compute Yes (SHARPv3 in NDR) No equivalent in standard Ethernet
Max fabric size (single subnet) ~48k endpoints (NDR), scaling with fat-tree Effectively unlimited via EVPN
Topology flexibility Fat-tree mostly; Dragonfly+ supported Any L2/L3 topology, including Clos and Dragonfly
Optical transceivers IBTA-tested, often proprietary form factors QSFP-DD / OSFP — broad multi-vendor supply
Software stack OFED, UCX, NCCL, MPI — all from one vendor ecosystem Same upper stack, but a wider choice of drivers and tuning knobs

Latency rows above are based on vendor datasheets (NVIDIA Quantum-2, Spectrum-4) and independent benchmarks from MLPerf and the TOP500 submission notes. Real numbers in production depend on switch buffer depth, MTU, and whether adaptive routing is enabled.

What the AI Training Workload Actually Demands

Distributed training is the workload that has pushed this comparison back to the top of the agenda. A single training step in a transformer model passes activations and gradients across all participating GPUs. For a 1024-GPU job with tensor parallelism and pipeline parallelism, the network must deliver:

  • Collective bandwidth on the order of TB/s across the job.
  • Tail latency for all-reduce operations on the order of single-digit microseconds.
  • Near-linear scaling efficiency: every doubling of nodes should give close to 2x throughput.

Three network features, more than any other, decide whether you hit those numbers:

1. RDMA, GPUDirect, and the Kernel-Bypass Path

RDMA lets one GPU’s memory be read or written by another GPU’s process without the data touching the host CPU or the kernel’s TCP/IP stack. NVIDIA GPUDirect RDMA goes one step further: the data goes directly between the GPU HBM and the network adapter’s DMA engine, bypassing the host entirely. This is the single biggest reason why Mellanox ConnectX-7 / ConnectX-8 adapters show up in nearly every modern AI cluster, whether the fabric underneath is InfiniBand or RoCE v2 Ethernet.

2. In-Network Compute (SHARP)

InfiniBand’s Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) moves all-reduce and broadcast operations off the endpoints and into the switch ASIC. In a 1024-GPU cluster, SHARPv3 can reduce collective traffic by 60–80% and cut the step time accordingly. There is no exact Ethernet equivalent. Spectrum-X and Ultra Ethernet add parts of the picture (congestion-aware routing, packet spraying) but not a full SHARP-style compute offload.

3. Adaptive Routing and Congestion-Free Fabrics

When one path in a fat tree gets hot, the fabric should re-route around it — at packet granularity, not flow granularity. InfiniBand switches have done this for years. Ethernet vendors have caught up: Spectrum-4 supports per-packet adaptive routing, and the Ultra Ethernet specification now standardizes a similar feature across vendors.

Ethernet Has Caught Up — Mostly

If you walked into a 2022 cluster, the gap between InfiniBand and Ethernet on AI workloads was wide enough that the choice was simple. By 2026, the picture is more nuanced:

  • 400 Gb/s and 800 Gb/s Ethernet are now shipping in volume. NVIDIA’s Spectrum-4 and Broadcom’s Tomahawk 5 both deliver 51.2 Tb/s per chassis, with 64x 800G ports. You no longer pay a speed tax for picking Ethernet.
  • RoCE v2 with proper PFC, ECN, and DCQCN tuning is a mature stack. We routinely see customers hit >95% of NCCL all-reduce bandwidth on well-built Ethernet pods.
  • NVIDIA Spectrum-X is a specifically engineered Ethernet platform for AI. It is not standard Ethernet — it adds adaptive routing, congestion control, and direct NCCL hooks — but it is Ethernet under the hood, with full IP/EVPN support alongside the AI-tuned path.
  • Ultra Ethernet (UE), ratified in 2024 and shipping in silicon from 2025 onward, is the industry attempt to standardize the Spectrum-X-style fabric across all Ethernet vendors. By mid-2026 there are working UE switch ASICs and adapter cards from at least three silicon vendors.

Rule of thumb in 2026: if your AI training cluster is larger than 256 GPUs and tightly coupled, you should still benchmark InfiniBand first. If you are in the 64–256 GPU range, or you have a mixed cloud+AI fabric, Ethernet (especially Spectrum-X or UE) is the easier choice — and the cost-per-port story is noticeably better.

Total Cost of Ownership: The Quiet Winner

Capex per port is the number most buyers start with, and on that metric Ethernet usually wins:

Cost Component InfiniBand (NDR) Ethernet (400G/800G)
Adapter (HCA / NIC), per port Higher (ConnectX-7/8 NDR) Lower (ConnectX-8 EN, BlueField-3, Broadcom NetXtreme)
Switch port, per 400G equivalent Higher (Quantum-2) Lower (Spectrum-4, Tomahawk 5)
Optics (400G/800G) IBTA-locked, narrower supply QSFP-DD / OSFP, multi-vendor
Cabling (copper / AECs) Limited, vendor-specific lengths Broad third-party supply
Power per 400G port (typical) ~12–15 W ~10–14 W
Software / management UFM / NetQ, included in stack Vendor NMS, plus SONiC / Spine-Leaf options
Training, hiring, skills Niche, harder to recruit Plentiful, mainstream skill set
Multi-vendor optics & switches No Yes

Where InfiniBand earns its premium is in operations and time-to-result. A cluster that comes up faster, runs cooler, and finishes a training run a few hours sooner pays back the fabric premium in months, not years. The TCO question is therefore workload-specific: a hyperscaler running always-on recommendation models amortizes fabric costs across many jobs and will choose Ethernet; a research lab running a single 8x H100 fine-tune of a 70B model will choose InfiniBand for the run-time savings.

What About Storage and HPC Workloads?

Not every cluster is an AI cluster. The other major network consumer is distributed storage:

  • NVMe-oF over RoCE v2 is the dominant protocol for high-performance block storage. It runs over Ethernet and is mature. NVMe-oF over InfiniBand exists but is rarely chosen — the customers who need it usually already have InfiniBand for compute and want a single fabric.
  • HPC MPI workloads (CFD, weather, genomics) have used InfiniBand for 20 years and still do, primarily because of the low-latency MPI collectives. Spectrum-X and UE are starting to displace InfiniBand in some HPC sites, but the migration is slow and depends on the MPI library (OpenMPI 5.x has better UE support than older releases).

Migration Paths and Hybrid Fabrics

You do not have to pick one fabric for the whole data center. Many sites we work with run a hybrid:

  • InfiniBand pod for the largest training jobs (256+ GPUs, NCCL-heavy).
  • Ethernet leaf-spine for everything else: inference, storage, development, management.
  • Border leaf connecting both fabrics at the data center gateway, with route filtering to keep failure domains separate.

ConnectX-8 adapters and BlueField-3 DPUs can present both an InfiniBand port and an Ethernet port from the same card, which makes the hybrid more practical than it used to be.

How Zhongxin Data Can Help

We supply both sides of this question and have done so for hundreds of cluster builds. Our typical engagement looks like this:

  1. Workload survey — what models, what batch sizes, what parallelism strategy, what storage traffic.
  2. Topology design — fat tree, Dragonfly+, or leaf-spine, with port counts and oversubscription ratios.
  3. Bill of materials — adapters, switches, optics, cabling, with a working BOM for both fabric choices.
  4. Rollout — pre-imaged racks, on-site racking and stacking, NCCL validation tests.

Browse the models listed on zhongxindata.hk: InfiniBand adapters and switches, Enterprise Ethernet networking gear, and RAID controllers / HBAs. If you are starting a new cluster and want a side-by-side BOM for InfiniBand vs Spectrum-X Ethernet, contact our solutions team — typical reply time is one business day.

Zhongxin Data is an independent supplier of enterprise and data center network hardware, headquartered in Hong Kong, China and shipping worldwide. NVIDIA, Mellanox, Broadcom, Intel, and the respective logos are trademarks of their owners. This article is for informational purposes and reflects market conditions as of mid-2026.

Contact us on WhatsApp +86 13068616914. Full specs and current availability are on zhongxindata.hk.


The decision framework we use with buyers

  • Start from the workload class — synchronous large-model training, inference serving and storage each stress a different part of the fabric
  • Then check the fabric you already have, because the switching cost often exceeds the hardware difference
  • Only then compare on port speed and latency, because those two numbers alone rarely decide the outcome

Matching hardware for whichever fabric you choose

  • The 100GbE switch tier and the 200GbE QSFP56 tier cover the Ethernet options; NDR InfiniBand covers the native-fabric option
  • ConnectX-class RDMA adapters cover both fabrics depending on the exact part number
  • Availability is confirmed per part number — send the model and quantity and we quote per part number

What we verify on the hardware we list

  • Every unit is cleaned, re-flashed and bench-tested on our own test line before it is listed
  • We confirm negotiated link rate, firmware revision and error counters against the part number you send
  • Ask us for photos of the actual unit before you commit


Related reading

Frequently Asked Questions

Should I use InfiniBand or Ethernet for a new AI cluster?

For a synchronous large-model training fabric where you want the lowest latency and the fewest fabric-management surprises, InfiniBand remains the safer default. For inference, storage and general-purpose cluster networking, Ethernet with RoCEv2 has reached the point where the economics usually favour it. If you already run one of them, staying is often cheaper than switching.

How large is the latency gap between them?

InfiniBand delivers consistently lower and tighter latency, particularly under load, where Ethernet depends on the lossless configuration being correct. The absolute difference is small enough that it only becomes the deciding factor in the most latency-sensitive training patterns — but the variance is what matters, because a stalled collective stalls the training step.

What is Ultra Ethernet and does it change the answer?

Ultra Ethernet is a consortium effort aimed at fixing the parts of Ethernet that make lossless AI traffic hard to tune, particularly congestion control and multipath. It should reduce the configuration burden that RoCEv2 deployments carry today. It does not change the InfiniBand engineering model, so the decision for new fabrics still turns on workload and installed base.

Can I migrate from one fabric to the other?

Partially, and it is worth planning. The host-side software path is portable because RDMA abstractions sit above the transport, but the switch fleet, the NIC population and the fabric management tooling all change. A common path is to run Ethernet for the build-out and keep an InfiniBand island for the most demanding training job until the Ethernet fabric proves out.

Which parts do you supply for either fabric?

We supply Mellanox switch and adapter families that cover both the Ethernet tiers and the InfiniBand option, and availability is confirmed per part number. Send the part numbers and quantities and we quote per part number with a condition note on the actual units.