Introduction: When the Fabric Breaks, the Cluster Stops

In modern AI data centers, the network fabric is no longer just plumbing—it is the critical infrastructure that determines whether a GPU cluster delivers exaFLOPS or sits idle. A single flapping port on an NVIDIA SN5600 switch can cascade into a collective operation timeout that stalls an entire training run costing thousands of dollars per hour. As fabrics scale to tens of thousands of endpoints at 400G and 800G, the traditional “ping and traceroute” approach to network troubleshooting is woefully inadequate.

This field guide distills the troubleshooting methodologies used by data center network engineers operating large-scale InfiniBand and Ethernet fabrics. We cover the diagnostic toolchains, common failure modes, and systematic approaches to isolating faults in production AI clusters. Whether you operate a small HPC cluster with Mellanox ConnectX-7 adapters or a hyperscale AI training fabric with NVIDIA Quantum-2 InfiniBand switches, these techniques will help you reduce mean-time-to-resolution (MTTR) and keep your GPUs fed with data.

Key Insight: In AI training fabrics, a 0.1% packet loss on a single rail can reduce collective operation throughput by 40% or more. Silent degradation is the enemy—your monitoring must catch problems before they manifest as slow jobs.

The Troubleshooting Toolkit: Layer by Layer

Effective network troubleshooting in data center fabrics requires tools at every layer of the stack. The table below maps common symptoms to the diagnostic tools and the OSI layer where the fault typically resides.

Symptom Likely Layer Primary Diagnostic Tools Typical Root Cause
Port down / link not established L1 (Physical) mellanox_burn, ethtool, mlxlink, optical power meter Dirty fiber, damaged transceiver, cable bend radius, module compatibility
Intermittent CRC errors L1 (Physical) ethtool -S, mlxlink, BER counters Marginal optics, connector contamination, EMI, temperature drift
High latency on specific path L2/L3 (Data/Network) perfquery (IB), show counters, ibdump, RoCE counters Congestion hotspot, ECMP hash polarization, PFC deadlock
Collective operation timeout L4/Transport NCCL debug logs, ib_write_bw, osu_benchmark, SHARP telemetry SHARP aggregation node failure, adaptive routing misconfiguration, silent packet drop
Training job hangs (all-reduce stall) L7/Application NCCL NCCL_DEBUG=INFO, nsys profiler, DCGM Rank imbalance, deadlocked dependency, NCCL topology mismatch

InfiniBand-Specific Tools

The InfiniBand ecosystem provides a rich set of diagnostic tools that have no direct Ethernet equivalent. Understanding these tools is essential for troubleshooting Quantum-2 and Quantum-3 based fabrics:

  • ibstat / ibv_devinfo: Verify port state (Active/Down/Initializing), link width, and speed. A port showing State: Active but Physical State: Polling indicates a link negotiation problem.
  • perfquery: Query performance counters on IB ports. Key counters include SymbolErrors, LinkErrorRecovery, PortXmitDiscards, PortRcvConstraintErrors. A rising LinkErrorRecovery counter signals marginal optics.
  • ibnetdiscover: Maps the entire fabric topology, showing every switch and endpoint with their LIDs. Critical for verifying cabling correctness and detecting topology mismatches after hardware swaps.
  • ibdiagnet: NVIDIA’s comprehensive fabric diagnostic tool that runs automated health checks across the entire subnet. Reports link quality, congestion, routing anomalies, and SM configuration issues.
  • mlxlink: Mellanox-specific tool for checking physical layer parameters—optical signal-to-noise ratio, pre-FEC BER, post-FEC BER, and eye diagram quality. Essential for diagnosing marginal transceivers.
  • saquery: Queries the Subnet Administrator for routing tables, path records, and multicast group information. Use it to verify that traffic is taking expected paths.

Ethernet / RoCE Tools

For Ethernet fabrics running RoCE v2 (used in NVIDIA Spectrum-X and Ultra Ethernet Consortium architectures), the toolset overlaps with traditional Ethernet diagnostics but adds RoCE-specific counters:

  • ethtool -S: Read NIC and switch port counters. Key RoCE counters: rx_roce_cnp (CNP packets), tx_roce_cnp, rx_roce_drops, rx_roce_pps. CNP (Congestion Notification Packets) count indicates active congestion control.
  • show tech-support: Dumps all switch counters, logs, and configuration in one command. NVIDIA Cumulus Linux and ONiX provide this for rapid escalation to vendor support.
  • mellanox_burn / cable diagnostics: Time Domain Reflectometry (TDR) on DAC cables to locate cable faults to the meter. Also checks transceiver EEPROM for authenticity and operating parameters.
  • PFC watchdog counters: show pfcwd stats reveals if PFC watchdog is disabling ports due to stuck queues—a common symptom of head-of-line blocking.

Common Failure Modes and Resolution Playbooks

1. Port Flapping: The Silent Killer

Port flapping—a link repeatedly going down and up—is one of the most destructive failure modes in AI fabrics. In an all-reduce operation, every GPU must participate; a single flapping port causes the collective to stall repeatedly, and NCCL will eventually timeout the entire training job.

Diagnostic steps:

  1. Check ethtool -S or perfquery for LinkDowned counter. If incrementing, the port is genuinely going down.
  2. Run mlxlink to check BER. A pre-FEC BER above 1E-4 on a 400G link suggests marginal optics or dirty connectors.
  3. Check transceiver temperature. NVIDIA SN5600 switches report per-port optical module temperature; values above 70°C indicate thermal issues.
  4. Inspect fiber connectors with a fiber inspection scope. Even invisible contamination can cause intermittent CRC errors at 400G.
  5. Verify DAC cable length and specifications. Active DACs beyond their rated length exhibit temperature-dependent flapping.
BER Range Link Health Recommended Action Expected MTTR
< 1E-12 Healthy No action needed. Monitor periodically. N/A
1E-12 to 1E-8 Marginal Clean connectors, check temperature, schedule replacement 30 min
1E-8 to 1E-4 Degraded Replace transceiver/cable immediately, reduce link speed if possible 15 min
> 1E-4 Critical / Flapping Port will likely go down. Isolate, replace optics, re-test 45 min

2. Congestion Hotspots and PFC Deadlock

In lossless Ethernet fabrics using PFC (Priority Flow Control) and ECN (Explicit Congestion Notification), a congestion hotspot can cascade into a PFC deadlock—a situation where ports on multiple switches are paused waiting for each other, creating a permanent traffic stall.

Symptoms:

  • Throughput drops to near-zero on multiple flows simultaneously
  • PFC pause frame counters spike (tx_pause_storms, rx_pause_frames)
  • NCCL reports timeout on all-reduce but individual ib_write_bw tests pass in isolation
  • PFC watchdog may or may not trigger depending on configuration

Resolution steps:

  1. Identify the stalled queue: Run show counters interface on each switch in the path. Look for tx_discards or egress_drops on a specific priority (usually priority 3 for RoCE).
  2. Trace the PFC storm: Use show pfcwd stats to find which ports are receiving PFC pause frames. The source of the pause is downstream of the congestion point.
  3. Check ECN configuration: Verify that ECN marking thresholds are set correctly. If ECN is too aggressive, CNP storms can cause throughput collapse. If too lenient, PFC triggers before congestion control reacts.
  4. Verify ECMP hash distribution: Use show ecmp to check if flows are polarizing onto a single link. This is common with fat-tree topologies when the hash seed is poorly configured.
  5. Check for deadlock recovery: NVIDIA Spectrum switches support automatic deadlock detection and recovery (port reset). Verify this is enabled: show pfcwd config.

Pro Tip: In NVIDIA Spectrum-X fabrics, the BlueField-3 DPU can perform per-flow congestion telemetry. Enabling telemetry streaming on the DPU gives you per-flow RTT and queue depth without affecting data plane performance.

3. InfiniBand SHARP Aggregation Failures

NVIDIA’s SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) offloads collective operations to the network switches. When a SHARP aggregation node fails, collective operations fall back to software-based all-reduce, causing a 2-5x performance degradation that is difficult to diagnose without understanding the SHARP architecture.

Symptoms:

  • NCCL reports all-reduce completion but latency is 2-5x higher than baseline
  • ib_write_bw point-to-point tests show normal performance
  • SHARP group creation fails or SHARP trees show incomplete membership
  • Job runs succeed but take significantly longer than expected

Diagnostic steps:

  1. Verify SHARP is enabled: Check NCCL environment variables: NCCL_IB_HCA=mlx5, NCCL_NET_GDR_LEVEL=PHB, NCCL_DEBUG=INFO. Look for “SHARP” in the NCCL log output.
  2. Check SHARP manager: The SHARP aggregation manager runs on the subnet manager (UFM/SM). Query sharp_am_get_info to verify the aggregation manager is running and has assigned trees.
  3. Inspect SHARP trees: Use sharp_coll_dump to dump the current SHARP tree topology. A broken tree (missing nodes) indicates a switch or endpoint that lost SHARP membership.
  4. Check switch SHARP capability: Not all switches support SHARP. Verify that the switches designated as aggregation nodes (usually spine switches) have SHARP-capable hardware (Quantum-2 SN5600 and later).
  5. Review UFM events: NVIDIA UFM logs SHARP-related events including tree rebuilds, aggregation node failures, and membership changes. Check the UFM event log for anomalies during the problem window.

4. Silent Performance Degradation: The Hardest Problem

The most insidious fabric problems are those where the network appears healthy—links are up, error counters are zero, point-to-point bandwidth tests pass—but collective operations are slower than expected. This “silent degradation” is common in large fabrics and requires correlation of multiple telemetry sources to diagnose.

Common causes of silent degradation:

Root Cause Detection Method Performance Impact Resolution
Adaptive routing misconfiguration Compare per-path latency with osu_latency; check show adaptive-routing on switches 10-30% collective throughput loss Reconfigure AR packet spraying; verify AR-capable switches
ECMP hash polarization Per-port throughput analysis shows uneven distribution (>20% variance) 15-40% effective bandwidth loss Adjust ECMP hash seed; enable per-flow entropy
GPU Direct RDMA (GDR) fallback Check NCCL log for “GDR” vs “bounce buffer”; nvidia-smi P2P topology 30-60% all-reduce throughput loss Verify PCIe topology, NCCL_NET_GDR_LEVEL, BAR1 size
NUMA imbalance numactl --hardware, lspci NUMA affinity, NCCL topology XML 10-25% cross-NUMA penalty Pin processes to correct NUMA node; use NCCL_IB_QPS_PER_CONNECTION
Firmware/driver version mismatch ibv_devinfo -v, mst version, compare across nodes Variable; can cause AR failures or GDR issues Standardize on tested firmware/driver combinations

Building a Proactive Monitoring Stack

The best troubleshooting strategy is preventing problems before they impact workloads. A modern data center fabric monitoring stack should include the following components:

  1. UFM (Unified Fabric Manager): NVIDIA’s fabric management platform for InfiniBand. Provides real-time topology visualization, performance monitoring, alerting, and automated remediation. Essential for Quantum-2/Quantum-3 fabrics.
  2. Cumulus Linux NetQ: Telemetry and observability for Ethernet fabrics running on NVIDIA Spectrum switches. Provides streaming telemetry, anomaly detection, and historical analysis.
  3. Prometheus + Grafana: Open-source monitoring stack. Collect switch and NIC counters via SNMP exporters or streaming telemetry (gNMI). Build dashboards for key metrics: BER, PFC pause frames, ECN marking rates, per-port throughput.
  4. NCCL Telemetry: NCCL can export per-collective-operation timing data. Correlating NCCL timing spikes with switch counter anomalies is the most effective way to catch silent degradation.
  5. DCGM (Data Center GPU Manager): While primarily a GPU monitoring tool, DCGM can detect GPU Direct RDMA failures and PCIe topology issues that manifest as network problems.

Escalation Matrix: When to Call Vendor Support

Not every problem can be solved with open-source tools. Knowing when to escalate to NVIDIA or your hardware vendor is critical for minimizing downtime. Use the following matrix as a guideline:

Scenario Self-Service Resolution Time Escalate When Support Tier
Port down, known cause (cable, transceiver) < 30 min Multiple ports down simultaneously, no common cause Tier 2 (switch vendor)
Congestion hotspot, single flow 1-2 hours PFC deadlock persists after reconfiguration Tier 2 (fabric specialist)
SHARP tree failures 2-4 hours SHARP manager crashes or trees cannot be rebuilt Tier 3 (NVIDIA UFM team)
Silent degradation, unclear cause 4-8 hours After exhausting diagnostic tools, correlated telemetry reviewed Tier 3 (vendor fabric engineering)

Best Practices for Reducing MTTR

Based on field experience operating large-scale AI fabrics, the following practices consistently reduce mean-time-to-resolution:

  • Maintain a known-good baseline: Run osu_benchmark and ib_write_bw across all node pairs when the fabric is healthy. When problems arise, compare against this baseline to immediately identify degraded paths.
  • Automate fabric health checks: Script daily runs of ibdiagnet or show tech-support with diff-based alerting. A change in counter values is more actionable than absolute thresholds.
  • Standardize firmware and driver versions: Version mismatches between switches, NICs, and the subnet manager are a leading cause of subtle interoperability issues. Maintain a tested compatibility matrix and enforce it during deployment.
  • Label and document cabling: In fabrics with thousands of cables, unlabeled connections turn a 15-minute cable swap into a 2-hour hunt. Use consistent labeling schemes and maintain a digital cable map.
  • Keep spare optics and cables on hand: A transceiver failure at 3 AM should not wait for procurement. Maintain a spares inventory proportional to fabric size (typically 5-10% of deployed optics).
  • Correlate NCCL logs with fabric telemetry: NCCL’s NCCL_DEBUG=INFO output contains timestamps and rank information that, when correlated with switch counter data, pinpoint the exact moment and location of problems.

Conclusion

Data center network troubleshooting for AI fabrics is fundamentally different from traditional enterprise networking. The scale, speed (400G/800G), and sensitivity of collective operations mean that problems invisible to traditional monitoring can cause catastrophic performance degradation. By building a layered diagnostic toolkit, understanding the unique failure modes of InfiniBand and RoCE fabrics, and implementing proactive monitoring, operators can keep their GPU clusters running at peak efficiency.

At Zhongxin Data, we supply the data center hardware that powers these fabrics—from NVIDIA Mellanox InfiniBand adapters and Quantum switches to Broadcom MegaRAID controllers and enterprise SSDs. Whether you are building a new AI training cluster or maintaining an existing fabric, Our range at zhongxindata.hk covers genuine and used data center components ensures you have the right hardware at the right price.

Explore our product categories:


A diagnostic order that finds faults fastest

  • Work the layers in order — physical, link, congestion, then application — because most reported fabric problems resolve before the application layer
  • Separate a link fault from a congestion fault before changing anything, because the symptom overlaps and the remedies do not
  • Keep a known-good spare of the adapter and switch families in the estate, and we confirm availability per part number

The hardware checks that settle most port problems

  • Error counters and the negotiated rate settle whether a link fault is physical, optical or configuration
  • A BER that climbs steadily points at the path — cable, optic or connector — rather than at the switch
  • We supply the switch and adapter families these diagnostics run against, and availability is confirmed per part number

What we verify on the hardware we list, before it reaches your rack

  • Every unit is cleaned, re-flashed and bench-tested on our own test line before it is listed
  • We confirm negotiated link rate, firmware revision and error counters against the part number you send
  • Ask us for photos of the actual unit and the firmware level before you commit


Related reading

Frequently Asked Questions

What is the most common cause of port flapping in high-speed fabrics?

A marginal physical path — a cable that is not seated properly, an optic operating at a temperature outside its rating, or a connector that has been handled and degraded. Flapping that tracks temperature points at the optic; flapping that follows a physical intervention points at the connection. The error counters settle it in minutes.

How do I diagnose PFC deadlock versus normal congestion in a RoCE fabric?

Look at the buffer occupancy and the pause frames rather than the throughput. A deadlock shows a head-of-line blocking pattern with pause propagating back toward the source, while normal congestion shows a gradual occupancy rise with the congestion notification counters incrementing. The two are handled in opposite directions — one needs the lossless profile changed, the other needs capacity.

Why does my collective run slower than point-to-point bandwidth suggests?

Because a collective is a tree of many simultaneous transfers, and it is sensitive to the slowest path and to tail latency rather than to average throughput. A point-to-point test uses the whole link in one direction, while a collective shares it across many flows. Test with a real collective at the real message size before concluding the fabric is at fault.

What BER threshold should trigger proactive cable replacement?

The practical trigger is a rising trend rather than an absolute number — a link whose error counters climb steadily will fail, and replacing it before the link drops avoids the outage. The absolute threshold depends on the optics and the FEC mode in use, and the vendor’s specification for the module is the reference.

How often should I run fabric diagnostics?

Continuously for the counters and error rates, and periodically for the full diagnostic sweep. The counters are free to collect and catch a degrading path before it drops; the fuller sweep is what confirms whether a degraded path is the cable, the optic or the connector.