Skip to main content

Data Parallelism

Data Parallelism allows you to speed up model training by using multiple GPUs. It is suitable for training involving large datasets or large batch sizes.

This is the most documented and easiest distribution method to implement.

Hybrid Parallelism (Data and Model)

Data Parallelism can be combined with Model Parallelism when the model to be trained is too large to fit in the memory of a single GPU. See the dedicated documentation page on hybrid parallelism.

Principle​

Data Parallelism involves replicating the model across all GPUs and splitting the data batch into mini-batches. Each GPU then processes a mini-batch for parallel training across all mini-batches.

Sub-batch in multi-GPU
Diagram of data batch distribution across 256 GPUs. Source.

In this case, the training process is as follows:

  1. The training script is executed in distributed mode across multiple processes (i.e., multiple GPUs). Each GPU:
    1. reads a portion of the data (mini-batch).
    2. trains the model using its mini-batch.
    3. computes gradients based on its local information (local gradients).
  2. The global gradients are calculated by averaging the local gradients returned by all GPUs. At this stage, it is necessary to synchronise the different processes.
  3. The model is updated on each GPU.
  4. The algorithm is repeated until the end of training.

Data Parallelism Representation of distributed training across 3 GPUs using the data parallelism method. Source.

Practical Implementation​

Resources​

  • The Optimised Deep Learning on Jean-Zay training offered by IDRIS covers, among other topics, training parallelisation, including data parallelism. More information is available here.