If you have ever watched an LLM training run crawl because the GPUs were starving for data, the bottleneck was almost never the matrix multiplies. It was the path between the GPUs, the network, and the storage. GPU Direct RDMA is the family of NVIDIA technologies that removes the CPU and the host memory from that path, letting data flow directly between GPU memory and a peer GPU, a NIC, or an NVMe drive. In a modern AI data centers, GPU Direct is no longer a nice-to-have. It is the reason an 8-GPU node trains at 700 GB/s of all-reduce instead of 60 GB/s.

This guide walks through what GPU Direct RDMA actually does under the hood, how the four GPU Direct flavors (P2P, Storage, RDMA, and Async) fit together, what hardware you need on the NIC side, and how to design the fabric so that GPU Direct keeps paying off as you scale from one node to one thousand. We close with the five questions buyers ask most often when they specify a Mellanox / NVIDIA ConnectX, BlueField, or Quantum switch for a new GPU cluster, and how those answers map to products we ship at Zhongxin Data.

Table of contents

What GPU Direct RDMA actually is

GPU Direct is a set of NVIDIA kernel and driver features that let a third-party device DMA directly into or out of pinned GPU memory. “Third-party device” here means any peer that can issue a PCIe or NVLink peer-to-peer transaction: a sibling GPU on the same root complex, a network interface card, or an NVMe controller with the right firmware.

Without GPU Direct, a typical GPU-to-GPU transfer across nodes looks like this:

  1. Source GPU writes tensor to host DRAM (through PCIe).
  2. Host CPU copies the buffer into a pinned socket buffer.
  3. NIC’s DMA engine reads that socket buffer and posts it onto the wire.
  4. Remote NIC writes into its host DRAM.
  5. Remote CPU copies the buffer into the destination GPU.
  6. Destination GPU reads the data from host DRAM (through PCIe again).

With GPU Direct RDMA, steps 1, 2, 4, and 5 collapse. The source NIC DMAs straight from GPU memory; the destination NIC DMAs straight into GPU memory. The CPU and the host memory buffer are still in the picture for control plane operations, but they are no longer on the data path.

The headline number: GPU Direct RDMA removes roughly 60–80% of the latency of an equivalent host-mediated transfer, and it nearly doubles the achievable streaming throughput because you stop bouncing the same buffer across the PCIe root complex twice.

The four flavors of GPU Direct

NVIDIA bundles four related features under the GPU Direct brand. They are independent but composable, and most production clusters use all four at once.

GPU Direct feature comparison
Feature Source Destination Typical use case Protocol
GPU Direct P2P GPU Peer GPU (same node) NCCL all-reduce inside one box NVLink, PCIe P2P, BAR
GPU Direct RDMA GPU Remote GPU (across node) Distributed training collectives InfiniBand RC, RoCE v2
GPU Direct Storage NVMe / parallel file system GPU Checkpoint / dataset streaming NVMe-oF, NFS, SCSI, DAOS
GPU Direct Async NIC / DPU GPU Kernel-bypass data placement into GPU DPDK, DOCA, GPUAsync

Two of these deserve a closer look because they are the ones that drive network design.

GPU Direct RDMA (cross-node collectives)

This is the one that consumes the entire InfiniBand or RoCE fabric. NCCL detects that the source and destination buffers both live in GPU memory, sees that the NIC supports GPU Direct, and issues an RDMA write that points directly at the destination GPU’s BAR-mapped memory. The wire transaction looks exactly like any other RC RDMA write; what is different is that the PCIe transaction on the remote end terminates inside the GPU, not inside system memory.

The two protocols supported are:

  • InfiniBand Reliable Connected (RC) — the default for HDR 200 Gb/s and NDR 400 Gb/s fabrics.
  • RoCE v2 (RDMA over Converged Ethernet) — the default for Spectrum-X and Ultra Ethernet fabrics, with PFC + ECN for lossless behavior.

GPU Direct Storage (dataset and checkpoint I/O)

GPU Direct Storage extends the same trick to NVMe and parallel file systems. Instead of staging a 100 GB checkpoint through the CPU and host memory, the DPU or NIC DMAs straight from the NVMe controller’s submission queues into the target GPU. NVIDIA has demonstrated end-to-end bypass pipelines from local NVMe into a single H100 at over 90% of the device’s peak bandwidth.

The fabric implications are subtle. GPU Direct Storage does not require a special network adapter on the GPU side, but it does require:

  • NVMe-oF target support on the storage array (or a parallel file system like Lustre / GPFS / BeeGFS / DAOS).
  • A NIC or DPU that can act as a PCIe peer and bypass the kernel (ConnectX-6 Dx and later, or BlueField-2 / BlueField-3).
  • Pinned GPU memory on the destination side, owned by the consumer CUDA stream.

Why bypassing the CPU matters at LLM scale

It is worth being precise about the cost of the host-mediated path, because at 70-billion-parameter scale the data path is the whole job.

Cost of one GPU-to-GPU all-reduce at H100 NVL, 8 GPUs per node
Path Latency (one hop) Sustained BW (8 GPUs) CPU cores burned
CPU-mediated (memcpy + send) ~22 µs ~60 GB/s 4–6
GPU Direct P2P (NVLink) ~3 µs ~900 GB/s 0
GPU Direct RDMA (NDR 400G) ~5 µs ~720 GB/s 0

What this table hides is the secondary cost: every CPU-mediated transfer leaves interrupts, context switches, and cache pollution behind. On an 8-GPU H100 node, doing all-reduce through the CPU costs roughly 30–40% of one socket’s throughput for the duration of the collective, and that overhead scales with model size. Removing the CPU from the data path is what gives modern NCCL ring and tree algorithms room to actually achieve their theoretical bandwidth.

Why this matters now: Mixture-of-experts models with hundreds of experts per layer are dominated by all-to-all collectives. Expert parallelism turns a previously compute-bound step into a network-bound step, and the difference between CPU-mediated and GPU Direct RDMA on an all-to-all is the difference between a usable training run and an unusable one.

Hardware requirements: NICs, switches, PCIe, GPUs

GPU Direct is not magic. There are five pieces of hardware that all have to line up. Get any one of them wrong and you silently fall back to the slow path.

  1. NVIDIA / Mellanox ConnectX or BlueField NIC. ConnectX-5 supports GPU Direct RDMA over IB and RoCE. ConnectX-6 Dx adds hardware-accelerated storage and is the floor for new designs in 2026. ConnectX-7 adds 400 Gb/s and is the right pick for NDR fabrics. BlueField-2 / BlueField-3 add an Arm core that can host GPU Direct Async offloads.
  2. Matching switch fabric. For InfiniBand: Quantum-2 (HDR, 200 Gb/s) or Quantum-3 (NDR, 400 Gb/s) with managed subnet manager. For Ethernet: Spectrum-3 or Spectrum-4 with adaptive routing and RoCE congestion control.
  3. PCIe at Gen4 x16 or better. A ConnectX-7 running at PCIe Gen3 x16 is bandwidth-starved by roughly 35% relative to a Gen4 x16 slot. Gen5 is preferred for new builds; Gen6 unlocks 800 Gb/s adapters cleanly.
  4. GPU with large BAR support. A100, H100, H200, B100, B200, GB200 all expose enough BAR1 space for peer-to-peer DMA. Consumer cards (RTX 4090 and below) do not and will silently fall back.
  5. NUMA-aware host platform. The NIC must sit behind the same root complex as the GPUs that will DMA through it. Pinning NCCL to the wrong NUMA node is the single most common cause of “GPU Direct mysteriously doesn’t work” tickets.

Browse our Network Adapters collection for ConnectX-5, 6 Dx, 7, and BlueField-2 / BlueField-3 cards in both used condition, all flashed to the latest firmware and validated for GPU Direct.

Software stack: NCCL, UCX, OFED, DPDK

The hardware is the easy part. The software is where most teams lose a week the first time.

GPU Direct software stack
Layer Component What it does Notes
Application PyTorch / JAX / Megatron Issues collective ops Uses NCCL backend by default
Collective library NCCL Topology-aware rings / trees Auto-detects GPU Direct when peer buffers are pinned
Transport UCX / libibverbs / libxrc RDMA primitives UCX is the default for HPC, NCCL uses verbs directly
NIC driver / firmware MLNX_OFED User-space verbs, GPU Direct shim Match OFED version to NIC firmware exactly
Subnet manager (IB) OpenSM / UFM Routing, congestion maps Required for adaptive routing on Quantum switches
Subnet manager (Eth) Cumulus / SONiC + DCC PFC + ECN, QoS Required for lossless RoCE

Three rules of thumb keep this stack stable in production:

  • Pin the OFED version to the firmware version. Mixing MLNX_OFED 23.10 with NIC firmware 28.40 is the leading cause of subtle GPU Direct RDMA timeouts.
  • Run a managed subnet manager. OpenSM in stand-alone mode is fine for one rack. Once you go past two racks, run NVIDIA Unified Fabric Manager (UFM) so that adaptive routing and SHARP can coordinate across the fabric.
  • Test with NCCL all-reduce benchmarks before declaring victory. The nccl-tests all_reduce_perf binary will tell you in 60 seconds whether you are on the GPU Direct fast path or the slow path.

Sizing the fabric for GPU Direct

The easiest sizing mistake is to compute the bandwidth you need for a single all-reduce and forget that expert parallelism, pipeline parallelism, and checkpoint I/O all hit the same fabric at the same time.

A practical rule of thumb for a 2026 AI training cluster:

  • Per-GPU NIC bandwidth: 1× 400 Gb/s (NDR) or 1× 200 Gb/s (HDR) for the bulk of training jobs.
  • Per-GPU NIC bandwidth, frontier models: 2× 400 Gb/s on the rail-optimized hosts (front-end GPUs carry more cross-rack traffic).
  • Storage tier: a separate BlueField-3-connected storage VLAN, sized for 2× the checkpoint size in GB/s, so that checkpointing does not stall the next iteration.

For a 1,024-GPU cluster, this typically maps to a 2:1 oversubscribed fat-tree of Quantum-2 or Quantum-3 switches, with adaptive routing enabled and congestion control tuned for the expected mix of collectives.

Cost and refresh cadence

The honest cost story is that a properly sized GPU Direct fabric costs about the same as the GPUs it connects. A 1,024-GPU H100 cluster with NDR fabric will spend roughly an amount quoted per part number–an amount quoted per part numberper GPU per hour on the network alone, amortized over a four-year life. The good news is that the fabric is the most reusable part of the cluster: a 2023 NDR fabric will happily carry B200 and GB200 GPUs at full speed for several generations.

This is where used InfiniBand switches and ConnectX adapters start to make economic sense for capacity-tier clusters. We ship used Quantum-2 switches with verified firmware, full burn-in, and warranty terms confirmed per model, typically at 40–60% of the original-vendor list price. For inference clusters and capacity-tier training, this is the single biggest TCO lever in the design.

Rule of thumb: if a cluster is doing capacity training, inference, or model evaluation, used HDR/NDR fabric is the right answer. If it is doing frontier pretraining, new NDR/XDR fabric pays for itself in faster time-to-first-checkpoint.

If you are designing or refreshing a GPU cluster, these guides cover the rest of the stack:

Browse our Network Adapters for ConnectX-5 / 6 Dx / 7 and BlueField cards, InfiniBand Switches for Quantum-2 / Quantum-3 fabric, and Used Network Switches for capacity-tier builds. Need help sizing the fabric? Contact our solutions team on WhatsApp with your GPU count, model, and topology target.

Contact us on WhatsApp +86 13068616914. Full specs and current availability are on zhongxindata.hk.


The hardware the fast path depends on

  • The path needs an RDMA-capable adapter, a switch fabric that preserves lossless behaviour, and a GPU platform that supports the peer path
  • Adapter and switch choices here are the same families as the general RDMA ones, and availability is confirmed per part number
  • Send the GPU platform and the interconnect plan and we quote per part number

Confirming the fast path is actually active

  • Counter-level verification is the only reliable check — throughput numbers alone can look healthy while the traffic takes a bounce-buffer path
  • Check the per-GPU counter set for the peer operations, not just the aggregate bandwidth figure
  • If the counters show the wrong path, the usual causes are the adapter mode, the switch lossless profile, or driver version mismatch

How we test the adapters and switches in an RDMA path

  • Every unit is cleaned, re-flashed and bench-tested on our own test line before it is listed
  • We confirm negotiated link rate, firmware revision and error counters against the part number you send
  • Ask us for photos of the actual unit before you commit


Related reading

Frequently Asked Questions

Is GPU Direct RDMA the same as InfiniBand RDMA?

No. GPU Direct RDMA is the mechanism that lets a GPU DMA directly into host memory over an RDMA transport, bypassing the CPU bounce path. InfiniBand is one transport that can carry it; RoCEv2 on Ethernet is another. The GPU-side optimisation and the transport choice are separate decisions, and both are needed for the fast path.

Do I need InfiniBand to use GPU Direct RDMA?

No. It works over RoCEv2 on Ethernet as well. What matters is that the adapter supports RDMA, the switch fabric is correctly configured for lossless behaviour, and the platform supports the peer path. InfiniBand is one way to satisfy those requirements, not the only one.

Will GPU Direct work with consumer GeForce cards?

Generally not in the supported configuration. The peer-to-peer path used by GPU Direct is available on data centre GPU platforms; consumer cards typically lack the support, and where the peer operation is unavailable the traffic falls back through host memory, which costs the performance the technique exists to recover.

What is the difference between GPU Direct RDMA and GPU Direct Storage?

GPU Direct RDMA moves data between GPU memory and remote host memory over RDMA. GPU Direct Storage moves data between GPU memory and storage, bypassing the CPU. They solve different problems — one is the collective path inside a training job, the other is the data-loading path — and a cluster usually needs both before the storage path stops limiting the job.

How do I know if my cluster is on the GPU Direct fast path?

Read the per-GPU counters for the peer operations. An aggregate bandwidth number can look correct while individual transfers still traverse host memory, because the fallback still moves the same bytes. The per-operation counters are what distinguish the two paths.