NCCL

1. 1

   Rank discovery

   Communicator assigns ranks and builds a ring or tree topology.

2. 2

   Chunk buffers

   Split tensors into chunks sized for NIC and NVLink paths.

3. 3

   Reduce-scatter

   Each rank reduces its chunk ownership across the ring.

4. 4

   Allgather

   Reduced chunks circulate until every rank holds the full result.

5. 5

   Sync

   Barrier or stream sync before the next compute kernel.

Step 1 / 5

NCCL fields