Network topology for distributed training sets how fast gradients can move between GPUs every step, so the wiring between servers can limit training speed even when the chips are identical. The mechanism is well documented. The size of the effect in real runs is not, and most of the strongest numbers come from simulations or vendor-adjacent sources. This piece separates the two.
What an all-reduce asks of the network
In synchronous data-parallel training, each GPU computes gradients on its share of the batch, and an all-reduce sums them across all P participating GPUs so every replica applies the same update. According to the ATP preprint (arXiv 2301.08658), synchronous all-reduce of large tensors is required in each step and places high demands on interconnect bandwidth. With the ring algorithm, each GPU moves about 2(P-1)/P times the message size. The multi-node NCCL model paper (arXiv 2511.09557) and a secondary explainer on factryze.ai both state this formula. As P grows, that factor approaches 2, so per-GPU traffic does not shrink as you add GPUs.
Why the slowest link sets the pace
In a ring, data passes hop by hop, and each step waits on the slowest link it crosses. The ATP preprint argues that fast links inside a server go unused when a cross-server link sets the pace. In the sources reviewed here, that claim rests on this one preprint, though it is consistent with NCCL-style documentation.
Intra-node versus inter-node links
The NCCL model paper assumes intra-node links have lower latency and higher bandwidth than inter-node links. It models inference rather than training, so it supports the assumption without measuring a training job. A secondary NCCL explainer (ai-infrastructure.net) puts the same all-reduce at tens versus hundreds of GB/s depending on whether traffic crosses NVLink or PCIe. I did not check those figures against a primary source, so treat them as unconfirmed.
Ring versus tree: when the algorithm choice matters
NCCL primarily implements Ring and Tree all-reduce. Ring is bandwidth-optimal. Tree takes about 2·log2(P) hops and suits small messages (factryze.ai). A maintainer comment on NVIDIA's NCCL GitHub issue 471 describes a flat ring as high-latency with the best bandwidth, and a tree as low-latency with acceptable bandwidth. That comment may predate current NCCL behaviour, so read it as historical characterisation.
NCCL selects the algorithm per call from message size and topology, so no fixed node-count cutoff is an established rule. The sources disagree on where any crossover falls. A vendor blog (Spheron) says Ring is optimal from 2 to 32 nodes and Tree's latency advantage wins beyond 32, but its excerpt cites no benchmark. A personal blog by Jingchao Zhang measured Qwen 7B fine-tuning at 2 and 4 nodes and found the sign of the Ring-versus-Tree difference varied by configuration. That is one author and a small experiment. No source I retrieved settles the crossover.
Layouts: fat-tree, rail-optimized and other network topology options
A Clos network, usually called a fat-tree, connects leaf switches to spine switches, with pods interconnected through core switches (arXiv 2407.20018). That survey reports that Meta's cluster supporting up to 24,000 GPUs, used for Llama 3 405B training, had a 7:1 oversubscription ratio at the core layer. The survey cites Meta; I did not retrieve Meta's own paper, so treat the figure as secondhand.
Oversubscription and what it limits
An oversubscription ratio compares downward capacity at a switch layer with its upward capacity. By that definition, at 7:1 traffic crossing the layer can use only about one-seventh of the capacity offered below it when ports are busy. This is my derivation from the definition, not a measured result for that cluster.
Rail-optimized wiring and the NVIDIA attribution
In a rail-optimized design, GPU index k on every server connects to rail k. Most same-index traffic therefore stays within one rail and reduces load on the spine switches (arXiv 2407.20018). A rail-only variant that drops any-to-any connectivity has been proposed in an MIT CSAIL HotNets 2023 workshop paper. Two blogs attribute the rail recommendation to NVIDIA reference designs. I did not retrieve NVIDIA's documentation, so that attribution is secondhand. A separate Introl blog argues fat-trees give predictable performance regardless of which GPU pair communicates. That is a design argument, not a measurement.
Design cases that disagree
Fat-tree is not uniformly preferred. The SAKURAONE cluster paper (arXiv 2507.02124) evaluated reference architectures, chose one optimised for shortest-path GPU communication, and judged fat-tree suboptimal for AI workloads at scale. That is one design decision, not a controlled comparison.
How much speed is at stake
The most specific quantitative result is a simulation. Arnold, a topology-aware scheduler presented at NeurIPS 2025, reduced the maximum spread of communication groups by up to 1.67x in simulation (abstract, mlanthology.org). That is a communication-group metric, not end-to-end training throughput. A secondary overview page reports a 10.6% production performance gain; I could not verify it in the full paper.
The DHelix paper (arXiv 2411.15871) reports that cross-node communication's share of total communication time rises from 14.8% to 33% when scaling from 8,192 to 16,384 nodes. Its table excerpt is partial and its scaling assumptions are not fully visible.
Scheduling and placement without new hardware
A GPU can belong to both a data-parallel group and a pipeline-parallel group. The Arnold abstract (arXiv 2509.15940) states that perfect alignment for both is unachievable, so a scheduler must balance them. The secondary overview page says all-reduce tolerates extra hops somewhat well, while pipeline point-to-point traffic is more sensitive to crossing network boundaries. Placement is a trade-off, not a free gain.
What remains unsettled
- A controlled fat-tree versus rail-optimized comparison on identical GPUs and the same job. I found none.
- Current NCCL algorithm selection and the ring-tree crossover. Settling it needs primary NCCL documentation and benchmarks at several node counts.
- How much communication stays exposed rather than hidden by compute overlap. A 2026 simulation preprint (arxiv.org/html/2605.19169) assumes an uncongested network, which limits what it shows.
- Vendor-adjacent claims that the network is the bottleneck more often than the GPU at frontier scale. A Prompt20 blog states this as opinion; I found no measurement behind it.
For a cluster purchase or design review, ask in writing for the oversubscription ratio at each switch layer, the measured all-reduce bandwidth at your node count and message sizes, and whether the scheduler considers topology. Cluster size is also bounded by power, a separate constraint covered in how data center power limits AI cluster size.