Introduction: Why Liquid Cooling Is Now a Networking Problem

For most of the last forty years, “data center cooling” meant computer room air conditioning (CRAC) pushing cold air under a raised floor. That world is gone. The arrival of 400 W–1000 W CPUs, 700 W–1200 W GPUs, and 800 Gb/s switches has made per-rack power density climb from 10 kW to 60 kW, 100 kW, and even 200 kW+ in flagship AI clusters. Air can no longer carry that heat away economically, and the bottleneck is no longer the server — it is the network fabric wrapped around the server.

This article is a field-tested guide to deploying liquid cooling together with a high-density network fabric. We cover the technology options (cold plate, single-phase immersion, two-phase immersion), the impact on top-of-rack switches, InfiniBand adapters, DPU-equipped BlueField SmartNICs, and 800G optical transceivers, and the operational practices that separate a pilot from a 50 MW AI data centers.

The new rule of thumb: if your rack power exceeds 30 kW, treat liquid cooling as a network project — because the heat removal path determines which NICs, switches, transceivers, and optical modules you can deploy.

1. Cooling Technologies Compared

There are three mainstream liquid-cooling approaches in 2026. Each has different implications for the network hardware installed in the same rack.

Technology Coolant Touches Rack Density Network Hardware Compatibility Typical Use Case
Air cooling (rear-door or in-row) None — fans + coils 10–30 kW All standard switches/QSFP-DD modules Legacy enterprise, low-density edge
Direct Liquid Cooling (DLC) / cold plate CPU, GPU, sometimes DIMMs and VRMs 40–80 kW ToR switches & NICs still air-cooled (front of rack) AI training, mainstream hyperscale
Single-phase immersion Entire server submerged in dielectric fluid 60–150 kW Switches require special immersion-rated models or stay in adjacent air-cooled rack Dense GPU pods, crypto/HPC farms
Two-phase immersion Entire server; fluid boils at component surface 100–250 kW Most switches NOT rated; pods are GPU-only Frontier-class HPC, prototype AI data centers

The table above is not theoretical — it is the practical compatibility matrix that drives procurement. A cold-plate DLC deployment can still use a standard NVIDIA Quantum-2 QM9700 or Spectrum-4 SN5600 ToR switch sitting at the top of the rack, because the switch’s heat-generating ASIC and PSUs are air-cooled. In a two-phase immersion tank, the same switch cannot be submerged; it must live in a “cold aisle” rack one or two positions away, with longer DAC/AEC cables bridging to the immersed GPU sleds.

2. The Network Stack Under Liquid Cooling

A typical 2026 AI rack looks like this when laid out for DLC cooling:

Layer Hardware Cooling Path Typical Power Critical Procurement Note
Compute sled 8x H200/MI300X GPU + dual EPYC Turin / Granite Rapids Cold plate on CPU, GPU, DIMMs ~10–14 kW Verify cold-plate covers all DIMM slots, not just CPU
Network sled ConnectX-8 / BlueField-3 DPU per GPU sled Cold plate or air (model-specific) 25–35 W per port BlueField-3 runs hotter than ConnectX — check vendor thermal spec
ToR switch QM9700 (NDR) / SN5600 (800G) Forced air (front-to-back) ~1.5–2.2 kW Airflow direction MUST match rack hot/cold aisle; check “port-side intake” vs “port-side exhaust”
Spine switch CS9500 / SN5600 in spine row Forced air ~2.5–3 kW Separate hot/cold aisle row; 12–15 kW per spine rack
Optical transceivers OSFP 800G DR8 / 2xFR4 Air (at switch faceplate) ~16–20 W each High-density OSFP cages need 30+ CFM of directed airflow

The crucial insight: in a DLC rack, the compute is liquid-cooled but the network is still mostly air-cooled. Your cooling design must handle both heat streams — a 14 kW compute sled dumps heat into the facility water loop, while the adjacent ToR switch exhausts ~2 kW into the rack’s air stream. Mixing them carelessly creates thermal dead zones and forces the ToR fans into a high-RPM state that accelerates failure.

3. Coolant Distribution Units (CDUs): Sizing for AI Loads

A CDU is the heart of a DLC loop. It isolates the facility water (which may be 30 °C warm water from a cooling tower) from the IT coolant (which touches chips at 45–65 °C). Sizing the CDU wrong is one of the most common causes of AI cluster underperformance.

CDU Class Loop Capacity Typical Deployment Network Considerations
Sidecar / rack-level 40–80 kW Single AI rack None — self-contained
Row-level 200–600 kW 4–12 racks in a hot/cold aisle Plumbing above rack, restrict top-of-rack optical cabling
Facility-level 2–20 MW Full AI data centers pod (50–200 racks) Manifold routing dictates aisle width; plan 1.2 m+ between hot/cold aisles

Two practical points from the field:

  1. Leave 20–30% thermal headroom in the CDU. AI workloads spike. A CDU sized to “average” rack power will throttle when eight adjacent racks run a collective-comms all-reduce at the same time. This is also when your SHARP / NCCL traffic peaks, which is precisely when you cannot afford thermal throttling on the ToR switch.
  2. Coolant temperature affects switch inlet air. Modern DLC designs run IT coolant at 40–50 °C. The “cold air” entering the ToR switch is therefore 35–45 °C, not the 20 °C of legacy data halls. Verify that your switch model is rated for an ASHARE A4 (45 °C) inlet — many older ToR models are only A3 (40 °C) and will derate throughput.

4. High-Density Rack Topology Considerations

Once you cross 50 kW per rack, the physical layout of the network becomes a first-class design constraint. Three principles have emerged from real deployments:

  • Keep ToR within 2 m of the GPU sleds. Beyond that, DAC/AEC cable reach becomes the limiting factor at NDR 400G speeds, and AOC fiber costs balloon. This is why immersion-cooled pods often use an adjacent “switch rack” pattern rather than co-locating the switch in the tank.
  • Plan for 1U or 0U ToR, not 4U chassis. Modern AI fabrics favor fixed-configuration 1U 64-port ToR switches (QM9700, SN5600) over modular chassis. They consume less rack space, have shorter airflow paths, and can be replaced unit-by-unit during maintenance without disrupting an entire rack.
  • Optical cabling must be serviceable from the cold aisle. With 200+ fibers per rack, MPO/MTP trunk density becomes unmanageable from the rear. Spec “port-side intake” switches so all patching happens at the cold aisle face.

5. Operational Realities: What Trips Up First

Across multiple AI data centers builds, the failure modes that hit first are predictable:

Issue Root Cause Network Impact Mitigation
ToR switch fans running at 100% RPM Inlet air too warm (A3 vs A4 inlet spec) PFC pause storms under thermal throttle Specify A4-rated switches; raise cold-aisle setpoint
Intermittent 800G optical link flaps OSFP cage thermal runaway Collective-comms timeouts, job restarts Increase directed airflow to faceplate; re-seat modules
Coolant leak trips IT-side loop Quick-disconnect failure on cold plate Entire rack goes dark; AI job loss Leak sensors + automatic isolation valves per rack
BlueField-3 DPU overheating DPU not on the cold-plate manifold Storage offload and OVS throughput drop 30–50% Confirm vendor DLC kit covers DPU; add heatsink if not

6. Used Hardware in Liquid-Cooled Racks

Many buyers assume that a DLC AI deployment must use brand-new switches and adapters. In practice, used ConnectX-6/7 adapters and Quantum-2 switches are deployed every day in cold-plate DLC racks, with no degradation in thermal performance. The key checks:

  • Used switches must be fitted with new fans and retested — old fan bearings fail faster at elevated cold-aisle temperatures.
  • Used transceivers must have fresh DOM (digital optical monitoring) calibration; aging laser bias currents drift outside spec after years of storage.
  • Adapter cards from previous-generation hosts (e.g. ConnectX-6 Dx on a PCIe Gen4 system going into a Gen5 DLC sled) work fine, but verify the host’s PCIe lane count — Gen5 x16 is required to feed 400G NDR at line rate.

If your procurement team is budget-constrained, a hybrid new/used approach — new ToR switches, used adapters and DAC cables — can deliver the same fabric performance at 40–60% of the all-new capex.

8. Conclusion: A Network-Aware Cooling Strategy

Liquid cooling is no longer optional for new AI data centers, but it is also not a stand-alone project. The ToR switch, the DPU, the optical transceiver, the rack airflow, and the CDU must be designed together. Buyers who treat cooling as “facilities” and networking as “compute” end up with stranded capacity and thermal bottlenecks at exactly the moments when collective-comms traffic peaks.

For data centers in the 40–80 kW range, direct liquid cooling with a row-level CDU and air-cooled ToR switches is the most flexible, lowest-risk path. Above 100 kW, immersion becomes attractive but forces the network into a separate physical layout. In either case, choose ASHRAE A4-rated switches, port-side intake airflow, and DPU-aware cold-plate coverage — and verify every component against your specific coolant inlet temperature.

For data centers in the 40–80 kW range, direct liquid cooling with a row-level CDU and air-cooled ToR switches is the most flexible, lowest-risk path. Above 100 kW, immersion becomes attractive but forces the network into a separate physical layout. In either case, choose ASHRAE A4-rated switches, port-side intake airflow, and DPU-aware cold-plate coverage — and verify every component against your specific coolant inlet temperature.

9. Heat Reuse and CDU Topology Choices

One under-discussed benefit of liquid cooling is that the heat it captures is useful. Facility warm water leaving a CDU typically runs 50–65 °C — hot enough for:

  • District heating networks — European operators (e.g. Equinix Paris, DataCenter Donauworth) pipe waste heat to nearby residential and commercial users, recovering 60–80% of the input energy.
  • On-site absorption chillers — for mixed-use campuses where office buildings still need chilled water, the GPU heat can drive an absorption chiller at surprisingly good COP (0.6–0.8) compared to electric chillers (COP 4–6 but with grid losses).
  • Agricultural and aquaculture — fish farms, greenhouse heating, and algae bioreactors can absorb low-grade heat that would otherwise be wasted.

From a CDU topology standpoint, three patterns dominate 2026 deployments:

Topology Heat Reuse Potential Network Rack Constraints Best Fit
Single-loop direct (no CDU) Low (facility-water only) None Pilot / under 200 kW total
Two-loop with row CDU Medium (warm water 40–55 °C) Top-of-rack plumbing restricts 1U ToR airflow Most common AI data centers, 1–10 MW
Three-loop with heat exchanger + facility CDU High (district-heating-grade 55–70 °C) Plumbing + heat exchanger room eats ~10% of floor space Sustainability-focused builds, 5 MW+

The network impact of topology choice is small but real: in a three-loop design, the secondary CDU and heat exchanger must sit outside the hot aisle, and the facility-water loop must run with sufficient pressure to feed the district heating network. This affects where you can place spine switches and aggregation routers — typically in a dedicated network-only row that is air-cooled independently from the GPU rows.

For data center operators looking to refresh their fabric without a full capex cycle, used ConnectX adapters, Quantum switches, and DAC cables from a trusted supplier can deploy cleanly into DLC racks when properly retested.

Explore related categories at Zhongxin Data

Need help sizing a CDU, validating switch thermal specs, or sourcing DLC-ready hardware that has been retested? Contact Zhongxin Data on WhatsApp +86 13068616914 or browse our full data center catalog.

Full specs and current availability are on zhongxindata.hk.


The hardware that behaves differently under liquid cooling

  • Switches and adapters need the same airflow check as any other device — the cooling method changes the rack, it does not change the port behaviour
  • Optical transceivers are the component most sensitive to the new thermal envelope, because they are rated for a specific inlet temperature
  • We supply the switch and adapter families used in these racks, and availability is confirmed per part number

Sizing and the parts of a deployment that decide cost

  • Direct-to-chip and rear-door heat exchangers serve different rack densities, and immersion changes the cabling rules entirely
  • The coolant loop, manifold and CDU are sized from the rack power, not from the network load
  • Send the rack power target and the topology you plan, and we confirm the hardware list per part number

Buying used hardware into a liquid-cooled rack

  • A used part still needs to be re-flashed and tested on our own test line before it goes near a coolant loop
  • The thermal envelope is tighter than in an air-cooled rack, so the condition of fans and thermal sensors matters more than usual
  • Ask us for photos of the actual unit and the firmware level before you commit


Related reading

Frequently Asked Questions

Can I air-cool an 800G ToR switch in a 100 kW rack?

Not credibly. The power density at 100kW is beyond what air cooling can move at the rack level, so a rack at that density is liquid cooled as a whole. A switch that lands in an air-cooled rack in a mixed estate is fine; a switch inside a 100kW rack needs the liquid-cooled build and the matching airflow direction.

Do ConnectX-8 and BlueField-3 DPUs need cold plates?

It depends on the sled position and the cooling architecture. In a direct-to-chip design, components in the compute sled are behind cold plates; in a rear-door heat exchanger design the whole rear of the rack is cooled and components sit in the airflow path. The part number has to match the chassis design, and that is a question for the platform vendor rather than a universal rule.

Is immersion cooling compatible with optical transceivers?

Generally not, and this is the main cabling difference in an immersion pod. Passive and active copper and conventional optics all need a dry path, so immersion designs route connections out of the fluid to a connection panel or use special sealed optic variants. This is the single biggest cabling change an immersion deployment forces.

What inlet temperature should I design for?

Cold-plate designs typically target a coolant supply in the high twenties Celsius with a warm return well above that, while immersion sits at the coolant’s own temperature. The point that matters for the network is the inlet temperature the transceivers see — many modules are rated to 70C ambient and will throttle or report faults above it.

How much does DLC add to a GPU server’s capex?

The cold plates, manifolds, hoses and CDU share their cost across every server in the rack, which is why the per-server figure falls sharply as density rises and rises sharply at low density. Below roughly 30kW per rack, direct-to-chip typically costs more per server than the air it replaces.