Multi-node training — what the fabric actually does

A single node with eight GPUs will take you a long way. When you outgrow it, a new component quietly becomes the most important part of your training job: the network between the nodes. Get the fabric right and adding hardware keeps making training faster. Get it wrong and you buy twice the GPUs to run 1.3× as fast. This is a tour of what the fabric is doing and why its details show up directly in your throughput.

Two tiers of interconnect

Distributed training lives on a hierarchy of links, and they are not close in speed:

• Inside a node — NVLink. GPUs in the same chassis talk over NVLink, a very high-bandwidth, low-latency fabric. This is where you want the chattiest communication to happen.
• Between nodes — the network. Once traffic leaves the box it crosses a NIC and a switch fabric. Even a fast network is an order of magnitude away from intra-node NVLink in bandwidth and latency.

Every scaling decision is really about keeping communication on the fast tier and minimizing what has to cross the slow one.

What collective operations do

Data-parallel training keeps a copy of the model on each GPU and, every step, has to average the gradients across all of them. That averaging is a collective operation — usually an all-reduce. The amount of data moved per step scales with the size of the model, and it happens on every step, so the fabric is exercised continuously, not occasionally.

Two properties of the network matter here:

• Bandwidth decides how fast the gradient tensors move. It sets the floor on how long each all-reduce takes.
• Latency decides the overhead of the many small messages that ring and tree algorithms exchange. At scale, latency is what erodes efficiency as you add nodes.

Well-built training loops overlap communication with computation — kicking off the gradient exchange for early layers while later layers are still doing backprop. Overlap hides a lot, but it can only hide what the fabric is fast enough to move in time.

Why an RDMA fabric, and why non-blocking

Commodity Ethernet moves gradients too, but two properties make dedicated RDMA fabrics — InfiniBand or RoCE v2 — the default for serious training:

• RDMA and GPUDirect. Data moves GPU-to-GPU across the network without detouring through host CPU and system memory on each hop. That cuts both latency and CPU overhead out of the critical path.
• Non-blocking topology. In a non-blocking fabric, every node can talk to every other at full line rate simultaneously — the switch layer is not oversubscribed. Oversubscription is the hidden tax that makes a network look fast in isolation and collapse when all nodes run an all-reduce at once, which is exactly what training does.

A "fast NIC" on an oversubscribed fabric is a benchmark trap. The number that matters is sustained all-reduce bandwidth with the whole cluster participating.

Scaling efficiency, honestly

Nobody gets perfect linear scaling. As you add nodes, the compute per GPU stays fixed but the communication grows, so the fraction of each step spent talking rather than computing creeps up. The practical questions are:

• Where does the curve bend? The node count at which adding hardware stops paying off depends on model size, batch size and — heavily — the fabric.
• Strong vs weak scaling. Splitting a fixed problem over more GPUs (strong) stresses latency; growing the problem with the cluster (weak) stresses bandwidth. Know which one you are doing.

A better fabric moves the bend to the right: you stay efficient across more nodes before diminishing returns set in.

Keeping the hierarchy fed

The winning pattern is topology-aware: put tensor-parallel shards — the most communication-heavy split — inside a node on NVLink, and reserve the inter-node network for the less frequent data-parallel gradient exchange. On whole bare-metal nodes you control that placement because you hold the entire node and its NVLink domain.

For jobs that want the fast tier to extend past eight GPUs, GB300 NVL72 places 72 GPUs in a single NVLink domain — an entire rack behaving like one very large node, with non-blocking InfiniBand for anything beyond it. TheAI Cloud runs both: whole B300 nodes stitched by non-blocking InfiniBand, and GB300 NVL72 rack units.

Do not forget storage

Distributed training reads a lot of data and writes checkpoints that can be large. Both should live on off-node storage, not the local disk, for two reasons: local disks are not shared across nodes, and on TheAI Cloud drives are wiped at termination — so a checkpoint that only exists on-node is a checkpoint you can lose. Stream data in and push checkpoints out to durable storage; the docs cover the patterns.

FAQ

Do I need InfiniBand or RoCE for multi-node training?

For communication-heavy training at scale, an RDMA fabric — InfiniBand or RoCE v2 — is strongly preferred. RDMA and GPUDirect keep gradient traffic off the host CPU, and a non-blocking topology lets every node run collectives at full rate at once. Plain Ethernet without RDMA works for loosely-coupled jobs but becomes the bottleneck for synchronous data-parallel training as node count grows.

What is a non-blocking fabric?

A network topology where all nodes can communicate at full bandwidth simultaneously, with no oversubscription in the switch layer. It matters for training because gradient all-reduce has every node talking at once — an oversubscribed fabric that benchmarks well point-to-point will choke under that all-to-all pattern.

Why keep tensor parallelism inside a single node?

Tensor parallelism exchanges activations between GPUs on every layer, which is the most bandwidth- and latency-sensitive communication in the job. Keeping those GPUs on the same NVLink domain inside one node keeps that traffic on the fastest link available. Spreading a tensor-parallel group across nodes forces it onto the slower network and throttles the step.

Why can't I just store checkpoints on the local disk?

Local disks are not shared between nodes, and on TheAI Cloud they are wiped when the instance terminates — including at term end and at zero balance. A checkpoint that exists only on a node's local drive is lost when that node goes away. Write checkpoints to durable off-node storage so a restart or a new node can resume.