What Is InfiniBand and Why Does It Matter for AI Clusters?
InfiniBand is a high-performance, low-latency network architecture originally developed for clustering supercomputers. In modern AI training infrastructure, InfiniBand has become the de facto standard because it delivers 200 Gbps (HDR) bandwidth with sub-microsecond latency—critical characteristics for distributed GPU training where thousands of GPUs must synchronize model parameters every training step.
Unlike Ethernet (even RDMA over Converged Ethernet / RoCE), InfiniBand uses a credit-based flow control mechanism that guarantees zero packet loss at the transport layer. This eliminates TCP retransmission overhead and makes InfiniBand the preferred choice for NVIDIA DGX systems, HGX platforms, and large-scale AI supercomputers.
Key InfiniBand Components in AI Data Centers
| Component | Model | Port Speed | Ports | Typical Role |
|---|---|---|---|---|
| Network Adapter | NVIDIA ConnectX-6 HX | HDR 200Gb/s | 1-2 | GPU server NIC |
| Network Adapter | NVIDIA ConnectX-7 | NDR 400Gb/s | 1-2 | Next-gen GPU server NIC |
| Network Switch | NVIDIA SN3700 | HDR 200Gb/s | 64 | Leaf/spine switch |
| Network Switch | NVIDIA SN3800 | NDR 400Gb/s | 8+16 | Core/aggregation switch |
Performance Comparison: InfiniBand vs Ethernet for AI
| Metric | InfiniBand HDR | 100GbE (RoCEv2) | 25GbE (TCP) |
|---|---|---|---|
| Raw Bandwidth | 200 Gbps | 100 Gbps | 25 Gbps |
| Effective Throughput | ~190 Gbps | ~85 Gbps | ~22 Gbps |
| Latency (RDMA write) | ~0.6 us | ~2-3 us | ~50-100 us |
| Packet Loss | Zero (credit-based) | PFC-based | TCP retransmission |
| NCCL AllReduce (8 GPUs) | ~35 GB/s | ~18 GB/s | ~4 GB/s |
Frequently Asked Questions
Can I mix InfiniBand and Ethernet in the same cluster?
Yes. Many deployments use InfiniBand for the GPU training fabric and Ethernet for management, storage (NFS), and external access. NVIDIA ConnectX adapters support dual-port configurations where one port runs InfiniBand and the other runs Ethernet simultaneously.
Do I need new InfiniBand hardware, or can I use refurbished?
Refurbished InfiniBand hardware (ConnectX-5/6 adapters, SN3700 switches) from reputable suppliers like Zhongxin Data offers identical performance specifications at 40-60% lower cost. Each unit undergoes full diagnostic testing including link-layer stress tests, firmware verification, and port integrity checks.
How many switches do I need for a 256-GPU cluster?
A 256-GPU cluster (32 DGX servers) typically requires 4 leaf switches and 2 spine switches (6 SN3700 total). Each leaf switch connects to 8 servers, and the 2 spine switches provide full bisection bandwidth.
Zhongxin Data supplies tested NVIDIA InfiniBand hardware. Browse our network adapters and network switches.