Blog #pytorch #machine-learning #ai
PyTorch DDP for Fine-Tuning Gemma 4 on Multiple GPUs
How PyTorch DistributedDataParallel works and when it is the right tool for LoRA fine-tuning Gemma 4, with a complete torchrun script, memory math for the 12B model, and the failure modes that hang multi-GPU jobs.
- Published
- Updated
- Reading
- 28 min
- Author
- Sudhanva Narayana
Some time ago I built a small Megatron Parallelism Visualizer to illustrate the three classic ways to split a training job across GPUs (tensor, pipeline, and data parallelism). Tensor and pipeline parallelism get most of the attention because they make it possible to train a 70B-parameter model at all. Yet most fine-tuning jobs I see in practice don’t need either of them; they just need data parallelism done well.
This post covers PyTorch DistributedDataParallel (DDP) applied to LoRA fine-tuning of Gemma 4 on a single multi-GPU node. We’ll walk through how DDP overlaps gradient communication with backward computation, calculate the exact per-GPU memory budget for Gemma 4 12B, step through a complete torchrun script you can adapt directly, and catalog the failure modes that turn multi-GPU jobs into silent hangs.
Table of contents
- When DDP is the right first tool
- Gemma 4 facts that matter for training
- How DDP works
- Batch size and learning rate
- Memory budget for Gemma 4 12B with LoRA
- The training script
- Launch with torchrun
- Run it on Modal
- Failure modes that hang or corrupt a DDP job
- NCCL and debugging environment variables
- Measure throughput and scaling
When DDP is the right first tool
The visualizer’s README summarizes each of the three strategies in a single line:
- Tensor parallelism (TP) splits individual weight matrices across GPUs so each GPU computes a partial matmul before communicating to reassemble the activation.
- Pipeline parallelism (PP) places different layers on different GPUs and streams micro-batches through them, trading memory capacity for pipeline “bubble” overhead.
- Data parallelism (DP) gives every GPU a full copy of the model and a disjoint slice of the batch, averaging gradients with an all-reduce at the end of the step.
Megatron-Core combines all three (“3D parallelism”) because at frontier pretraining scale no single strategy is enough. Fine-tuning a model on a single eight-GPU node comes down to whether one full replica of your training state fits on one GPU.
flowchart TD
Fit{"One replica fits<br/>on one GPU?"}
Fit -->|Yes| DDP["DDP<br/>copy model, split data"]
Fit -->|No| Shard{"Only optimizer state<br/>and grads too big?"}
Shard -->|Yes| FSDP["FSDP / ZeRO<br/>shard the state"]
Shard -->|No| TPPP["Tensor / pipeline<br/>parallelism + DP"]
DDP -.-> Shrink["Almost fits? LoRA, QLoRA,<br/>checkpointing, shorter seqs"]
With LoRA, the answer for most Gemma 4 sizes is yes. The frozen base weights stay in bf16, the trainable adapter adds only tens of millions of parameters, and gradient checkpointing keeps activation memory bounded. Whenever a replica fits on a single GPU, DDP is the simplest and fastest way to scale across more GPUs. Nothing inside the model architecture changes, the only collective is a bucketed gradient all-reduce, and every rank runs identical code.
When a single replica doesn’t fit, you have other options, each tailored to a specific bottleneck:
| Strategy | What it splits | Use it when | Cost |
|---|---|---|---|
| DDP | The data | A full replica fits on one GPU | One gradient all-reduce per optimizer step |
| FSDP / ZeRO | Parameters, gradients, optimizer state | Full fine-tuning, where optimizer state dwarfs the weights | All-gathers in forward and backward, reduce-scatter of grads |
| Tensor parallelism | Individual weight matrices | Single layers are too large, or you need lower per-step latency | Collectives inside every layer, needs fast intra-node links |
| Pipeline parallelism | Layers | The model’s depth doesn’t fit, especially across nodes | Pipeline bubbles, micro-batch scheduling complexity |
For full-parameter fine-tuning of Gemma 4 12B, that table points straight to FSDP rather than DDP, and the memory budget section below shows the exact math. For LoRA on the same model, DDP is the right tool.
Gemma 4 facts that matter for training
Google released the initial Gemma 4 family in four sizes on March 31, 2026, followed by a 12B checkpoint on June 3, 2026. Pulling from the Gemma 4 model card and model overview, here are the specs that matter when sizing a training job:
| Model | Parameters | Layers | Context | Input modalities | BF16 inference memory (Google) |
|---|---|---|---|---|---|
| E2B | 2.3B effective, 5.1B with embeddings | 35 | 128K | Text, image, audio | 11.4 GB |
| E4B | 4.5B effective, 8B with embeddings | 42 | 128K | Text, image, audio | 17.9 GB |
| 12B | 11.95B | 48 | 256K | Text, image, audio | 26.7 GB |
| 26B A4B | 25.2B total, 3.8B active (MoE) | 30 | 256K | Text, image | 57.7 GB |
| 31B | 30.7B | 60 | 256K | Text, image | 69.9 GB |
Key details from the primary release docs:
- License. Google releases the weights under Apache 2.0, as documented in the model card, the Gemma 4 launch post, and the
google/gemma-4-12B-itHugging Face repo. - Model IDs. The base checkpoints are
google/gemma-4-E2B,google/gemma-4-E4B,google/gemma-4-12B,google/gemma-4-26B-A4B, andgoogle/gemma-4-31B, with-itsuffixes for the instruction-tuned variants. - Tooling. Day-one support landed in Hugging Face Transformers, PEFT, and TRL. Google’s own Gemma QLoRA fine-tuning guide pins
transformers>=5.10.1andpeft>=0.19.0to train an adapter on a single 16 GB T4. - Architecture. Every size uses a 262K vocabulary and interleaves sliding-window attention layers with global attention layers. The 12B model is encoder-free. Image and audio tokens feed directly into the unified decoder rather than through separate modality towers.
Inspecting the published safetensors index and config for google/gemma-4-12B-it surfaces three more architectural details that directly affect a DDP script:
- 11.96B parameters stored in BF16, with 11.91B living under
model.language_model. Because of the 262,144-token vocabulary and 3,840 hidden dimension, the tied input/output embedding matrix alone accounts for ~1.0B parameters (262144 × 3840). - Eight of the 48 layers use global attention (every sixth layer) with
attention_k_eq_venabled, so those layers definek_projand omitv_projentirely. Passingv_projin your LoRA target list still works cleanly; PEFT attachesv_projadapters to the 40 sliding-window layers that have one. - LoRA at rank 16 across all attention and MLP projections of the text decoder totals 65.6M trainable parameters (~0.55% of the model), calculated directly from the checkpoint tensor shapes.
Throughout this post I use google/gemma-4-12B-it. At 12B parameters, bad memory math will OOM your GPUs, while a bf16 LoRA replica still fits on a single 48 GB or 80 GB card.
How DDP works
The data-parallel simulation in my visualizer (simulate_data_parallel in simulator.py and dataParallel in simulator.js) shows the core loop. Every GPU runs forward and backward on its own shard of the batch across all gradient-accumulation micro-steps, followed by a single all-reduce that averages gradients before every replica takes an identical optimizer step.
Production PyTorch DDP follows the same structure and overlaps gradient communication with backward computation. Here is the sequence documented in the PyTorch DDP design notes and API reference:
1. One process per GPU
torchrun spawns one Python process per GPU and populates RANK, LOCAL_RANK, and WORLD_SIZE in the environment. Each process binds itself to its assigned GPU, joins a shared NCCL process group, and runs collectives over NVLink/PCIe within a node or over InfiniBand/RoCE across nodes.
2. Replicas start identical
Wrapping a module in DistributedDataParallel broadcasts parameters and buffers from rank 0 to every other rank so all replicas start from identical base weights and LoRA matrices. During construction, DDP also attaches an autograd hook to every parameter with requires_grad=True. Frozen base weights receive no hook and never touch the network, which is why LoRA and DDP pair so well.
3. The sampler shards the data
DistributedSampler partitions dataset indices across ranks using a deterministic seed combined with the epoch number. Always call sampler.set_epoch(epoch) at the top of your epoch loop; if you forget, every epoch replays the exact same shuffle order.
By default, the sampler pads the index list so every rank receives an identical batch count (or trims the tail when drop_last=True). The batch count must match across all ranks. An off-by-one batch count on a single rank hangs the job at the end of the epoch (see failure modes).
4. Backward overlaps with communication
While the visualizer shows forward, backward, and all-reduce as strictly sequential blocks, real DDP groups trainable parameters into buckets (default bucket_cap_mb=25) ordered roughly in reverse layer order to match backward execution. As soon as every gradient in a bucket is ready, DDP fires an asynchronous NCCL all-reduce for that bucket while autograd continues computing gradients for earlier layers. By the time backward reaches the first layer, most of the gradient communication has already finished in the background.
sequenceDiagram
participant G0 as GPU 0
participant G1 as GPU 1
participant NCCL as NCCL
Note over G0,G1: micro-batches 1..N-1 in no_sync()
G0->>G0: forward (last micro-batch)
G1->>G1: forward (last micro-batch)
G0->>G0: backward: last layers
G1->>G1: backward: last layers
G0->>NCCL: bucket k ready
G1->>NCCL: bucket k ready
Note over NCCL: all-reduce bucket k
G0->>G0: backward: earlier layers
G1->>G1: backward: earlier layers
G0->>NCCL: bucket 1 ready
G1->>NCCL: bucket 1 ready
NCCL-->>G0: averaged gradients
NCCL-->>G1: averaged gradients
Note over G0,G1: clip + optimizer.step()
DDP averages gradients across ranks rather than summing them. Because every rank starts from identical weights and applies the exact same averaged gradients in optimizer.step(), all replicas remain bit-for-bit identical without ever broadcasting weights again.
5. no_sync() for gradient accumulation
When using gradient accumulation, you only want to pay for an all-reduce on the final micro-batch before optimizer.step(), not on every intermediate micro-batch. Wrapping intermediate micro-batches in with model.no_sync(): suppresses the autograd reduction hooks so gradients accumulate locally in .grad; running the final micro-batch outside no_sync() triggers the bucketed all-reduce over the accumulated sum. Note that both the forward and backward pass must sit inside the no_sync() block; if forward runs outside it, DDP will still synchronize.
6. What DDP doesn’t do
DDP’s job ends once gradients are averaged. It does not average your logged loss across ranks, coordinate checkpoint writes, distribute validation loops, or save you if rank 0 branches into a collective that ranks 1–7 skip. All of those responsibilities fall on the training script.
Batch size and learning rate
Your effective (global) batch size is the product of three numbers:
global_batch = micro_batch_per_gpu × world_size × accumulation_steps
The Megatron visualizer expresses the same identity from the top down (micro_batch_size = batch_size // num_gpus and effective_batch_size = batch_size × gradient_accumulation_steps, verified in test_simulator.py where 128 samples across 2 GPUs with 4 accumulation steps yield an effective batch of 512). In a real training script, always compute global_batch bottom-up from micro_batch_per_gpu so integer division never hides dropped samples.
Keeping that identity in mind prevents two common multi-GPU mistakes:
- Adding GPUs changes the optimization dynamics if you hold accumulation fixed. Scaling from 1 GPU to 8 GPUs with the same per-GPU micro-batch and accumulation multiplier makes your global batch 8× larger and cuts the number of optimizer steps per epoch by 8×. Because DDP averages gradients across ranks, an 8-GPU DDP run is mathematically equivalent to a single-GPU run at that 8× global batch, so a learning rate tuned on one GPU won’t automatically transfer.
- Fix your target global batch size first, then solve for
accumulation_steps. If a single-GPU recipe converges well at a global batch of 64 (say,micro_batch=1andaccum=64), preserve that exact optimization trajectory on 8 GPUs by settingmicro_batch=1andaccum=8. Adding GPUs then shrinks wall-clock time without altering training dynamics.
While the classic linear scaling rule (scaling learning rate linearly with batch size after warmup) was derived for large-batch SGD on ImageNet, for AdamW LoRA fine-tuning it is best treated as a rough starting heuristic; always sweep the learning rate at the actual global batch size you plan to run.
Autoregressive language models introduce one more wrinkle: Hugging Face’s default causal-LM loss averages over the non-padding tokens within each micro-batch, and DDP then averages those per-rank losses with equal weight. If rank 0 happens to draw a 200-token sample while rank 1 draws a 2,000-token sample, each token on rank 0 gets 10× the gradient weight of a token on rank 1. For reasonably uniform chat datasets this rarely matters, but if your sequence lengths vary by an order of magnitude, compute a true global token average instead:
- Sum the unnormalized per-token loss on each rank.
all_reduceboth the loss sum and the active token count across ranks.- Divide by the global token count.
Memory budget for Gemma 4 12B with LoRA
Use this budget to decide which GPUs to request. To get the real peak on your hardware, call torch.cuda.max_memory_allocated() after the first few steps. GB means 10⁹ bytes.
The budget uses these assumptions:
google/gemma-4-12B-it- bf16 base weights
- LoRA rank 16 on all text-decoder projections (65.6M trainable parameters)
- Adapter weights in fp32 (PEFT’s default when the base model is bf16)
- AdamW
- Gradient checkpointing on
- Micro-batch 1
- Sequence length 2,048
| Component | Arithmetic | Size |
|---|---|---|
| Frozen base weights (bf16) | 11.96B × 2 bytes | 23.9 GB |
| LoRA weights (fp32) | 65.6M × 4 bytes | 0.26 GB |
| LoRA gradients (fp32) | 65.6M × 4 bytes | 0.26 GB |
| AdamW moments (fp32, two per param) | 65.6M × 8 bytes | 0.52 GB |
| DDP gradient buckets | 0 with gradient_as_bucket_view=True, else +0.26 GB |
0–0.26 GB |
| Checkpointed layer inputs | 2,048 tokens × 3,840 hidden × 2 bytes × 48 layers | 0.75 GB |
| One layer’s recomputed activations | MLP intermediates at 15,360 wide dominate | ~1 GB |
| Logits and loss | 2,048 × 262,144 = 537M values; bf16 logits, softcap temporaries, fp32 upcast for cross-entropy, and its gradient | ~6–9 GB |
| CUDA context, NCCL buffers, allocator slack | Varies by driver and fragmentation | ~2–4 GB |
| Total per GPU | ~35–40 GB |
The trainable state is tiny. The LoRA weights, gradients, and AdamW moments together take roughly 1 GB. Replicating 1 GB on every GPU is negligible, which is why DDP works so cleanly here.
The 262K vocabulary is the hidden activation hog. At 2,048 tokens, the output logits tensor holds 2,048 × 262,144 = 537M elements, and standard cross-entropy upcasts those logits to fp32 while Gemma 4’s final tanh logit softcap allocates additional temporaries. Doubling sequence length to 4,096 roughly doubles that single line item to 12–18 GB. If you hit an OOM at longer context lengths, look at the output head and cross-entropy loss before blaming the decoder layers (or reach for a fused chunked cross-entropy kernel).
Why the 16-bytes-per-parameter rule rules out DDP for full fine-tuning. As broken down in the ZeRO paper, mixed-precision AdamW requires ~16 bytes per trainable parameter (bf16 weights + bf16 gradients + fp32 master weights + two fp32 Adam moments). Across 11.96B parameters, that comes to ~191 GB of static state before a single activation is allocated, which is impossible on a single GPU. Full fine-tuning of Gemma 4 12B requires sharding that state across GPUs with FSDP or ZeRO, not replicating it with DDP.
Applying the same back-of-the-envelope math across the rest of the Gemma 4 lineup (bf16 base weights):
| Model | bf16 weights | DDP + LoRA outlook |
|---|---|---|
| E4B | 8.0B × 2 ≈ 16 GB | Fits 40–80 GB GPUs comfortably. On 24 GB GPUs, shorten sequences since the logits line is the same size as for 12B. |
| 12B | 11.96B × 2 ≈ 24 GB | Fits 48 GB GPUs at 2K tokens, 80 GB with headroom for longer sequences or micro-batch 2. |
| 26B A4B | 25.2B × 2 ≈ 50 GB | Memory scales with total parameters, not active ones. Tight on 80 GB. Consider QLoRA or FSDP. |
| 31B | 30.7B × 2 ≈ 61 GB | Too tight in bf16 on 80 GB once activations are added. QLoRA (4-bit base) per replica, or FSDP. |
On the 26B-A4B MoE checkpoint, while only 3.8B parameters are active per token for compute, all 25.2B parameters must reside in VRAM on every DDP replica. Furthermore, if you attach LoRA adapters to individual routed experts and a micro-batch happens not to route any tokens to one of those experts, DDP will complain about unused parameters unless handled explicitly.
Communication per optimizer step
Because DDP only synchronizes gradients for trainable parameters, LoRA slashes network traffic by the same ~99% factor as optimizer memory. A ring all-reduce transfers 2 × (N − 1) / N times the gradient buffer per GPU (the standard bus-bandwidth factor documented in nccl-tests, which equals 1.75× at N = 8 GPUs):
| What you train | Gradient buffer | Sent per GPU per step |
|---|---|---|
| LoRA r=16, text decoder (this post) | 65.6M × 4 bytes ≈ 0.26 GB | ≈ 0.46 GB |
LoRA plus trainable embed_tokens / lm_head (tied, ~1.0B params) |
+2.0 GB (bf16) to +4.0 GB (fp32) | ≈ 4–7.5 GB |
| Full fine-tune, bf16 gradients | 11.96B × 2 bytes ≈ 23.9 GB | ≈ 42 GB |
In the middle row, Google’s single-GPU QLoRA guide includes modules_to_save=["lm_head", "embed_tokens"] in its example config. On one GPU that’s just a VRAM trade-off, but under DDP unfreezing the 1.0B-parameter tied embedding table balloons the all-reduce payload by roughly 10×. Unless you are actively adding new tokens to the vocabulary, keep embed_tokens and lm_head frozen.
Dividing 0.46 GB by the bus bandwidth reported by nccl-tests on your node gives the raw communication time. At 100 GB/s of intra-node NVLink bus bandwidth, the entire LoRA all-reduce takes ~5 ms once per optimizer step, and most of that 5 ms is hidden behind backward computation.
The training script
Below is a complete, self-contained DDP training script using plain PyTorch DistributedDataParallel, Hugging Face Transformers, and PEFT. I skipped Trainer and TRL’s SFTTrainer here so every distributed primitive sits out in the open (launching TRL’s SFTTrainer under torchrun executes the same loop).
The dataset uses the same format as Google’s guide, with one JSON object per line:
{
"messages": [
{ "role": "user", "content": "..." },
{ "role": "assistant", "content": "..." }
]
}
# train_ddp.py: LoRA fine-tuning of Gemma 4 with plain PyTorch DDP.
# Launch: torchrun --standalone --nproc_per_node=8 train_ddp.py
import contextlib
import json
import os
import time
import torch
import torch.distributed as dist
from peft import LoraConfig, get_peft_model
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.utils.data import DataLoader, Dataset
from torch.utils.data.distributed import DistributedSampler
from transformers import (
AutoModelForCausalLM,
AutoTokenizer,
get_cosine_schedule_with_warmup,
)
MODEL_ID = os.environ.get("MODEL_ID", "google/gemma-4-12B-it")
DATA_PATH = os.environ.get("DATA_PATH", "train.jsonl")
OUTPUT_DIR = os.environ.get("OUTPUT_DIR", "checkpoints")
MAX_LEN = 2048 # tokens per sample after truncation
MICRO_BATCH = 1 # samples per GPU per forward pass
ACCUM_STEPS = 8 # micro-batches per optimizer step
EPOCHS = 1
LR = 1e-4
WARMUP_STEPS = 20
MAX_GRAD_NORM = 1.0
LOG_EVERY = 10
SEED = 42
# Only the text decoder's projections. Keeps LoRA off the vision/audio paths,
# which a text-only batch never touches.
LORA_TARGETS = (
r".*language_model.*\.(q_proj|k_proj|v_proj|o_proj|gate_proj|up_proj|down_proj)"
)
class ChatJsonl(Dataset):
"""One JSON object per line: {"messages": [{"role": ..., "content": ...}]}."""
def __init__(self, path, tokenizer, max_len):
with open(path) as f:
self.rows = [json.loads(line) for line in f if line.strip()]
self.tokenizer = tokenizer
self.max_len = max_len
def __len__(self):
return len(self.rows)
def __getitem__(self, idx):
text = self.tokenizer.apply_chat_template(
self.rows[idx]["messages"], tokenize=False
)
# The chat template already emits <bos>, so don't add it twice.
ids = self.tokenizer(
text, add_special_tokens=False, truncation=True, max_length=self.max_len
)["input_ids"]
return torch.tensor(ids, dtype=torch.long)
class PadCollate:
"""Right-pads a batch and masks padding out of the loss.
A top-level class, not a closure, so DataLoader workers can pickle it.
"""
def __init__(self, pad_id):
self.pad_id = pad_id
def __call__(self, batch):
longest = max(len(x) for x in batch)
input_ids = torch.full((len(batch), longest), self.pad_id, dtype=torch.long)
attention_mask = torch.zeros((len(batch), longest), dtype=torch.long)
for i, ids in enumerate(batch):
input_ids[i, : len(ids)] = ids
attention_mask[i, : len(ids)] = 1
labels = input_ids.masked_fill(attention_mask == 0, -100)
return {
"input_ids": input_ids,
"attention_mask": attention_mask,
"labels": labels,
}
def main():
# torchrun sets RANK, LOCAL_RANK, WORLD_SIZE, MASTER_ADDR and MASTER_PORT.
local_rank = int(os.environ["LOCAL_RANK"])
torch.cuda.set_device(local_rank)
device = torch.device("cuda", local_rank)
dist.init_process_group(backend="nccl", device_id=device)
rank = dist.get_rank()
world_size = dist.get_world_size()
is_main = rank == 0
torch.manual_seed(SEED) # same LoRA init on every rank
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
MODEL_ID, dtype=torch.bfloat16, device_map={"": local_rank}
)
model.gradient_checkpointing_enable(
gradient_checkpointing_kwargs={"use_reentrant": False}
)
model = get_peft_model(
model,
LoraConfig(
r=16,
lora_alpha=32,
lora_dropout=0.05,
target_modules=LORA_TARGETS,
task_type="CAUSAL_LM",
),
)
if is_main:
model.print_trainable_parameters()
model = DDP(model, device_ids=[local_rank], gradient_as_bucket_view=True)
torch.manual_seed(SEED + rank) # but different dropout masks per rank
dataset = ChatJsonl(DATA_PATH, tokenizer, MAX_LEN)
sampler = DistributedSampler(dataset, shuffle=True, seed=SEED, drop_last=True)
loader = DataLoader(
dataset,
batch_size=MICRO_BATCH,
sampler=sampler,
collate_fn=PadCollate(tokenizer.pad_token_id),
drop_last=True,
num_workers=2,
pin_memory=True,
)
trainable = [p for p in model.parameters() if p.requires_grad]
optimizer = torch.optim.AdamW(trainable, lr=LR, weight_decay=0.0, fused=True)
steps_per_epoch = len(loader) // ACCUM_STEPS # identical on every rank
scheduler = get_cosine_schedule_with_warmup(
optimizer, WARMUP_STEPS, steps_per_epoch * EPOCHS
)
if is_main:
global_batch = MICRO_BATCH * world_size * ACCUM_STEPS
print(f"world={world_size} global_batch={global_batch} steps={steps_per_epoch}")
for epoch in range(EPOCHS):
sampler.set_epoch(epoch) # new shuffle each epoch, same on all ranks
model.train()
batches = iter(loader)
for step in range(steps_per_epoch):
t0 = time.perf_counter()
loss_sum = torch.zeros((), device=device)
tokens = torch.zeros((), device=device)
for micro in range(ACCUM_STEPS):
batch = {
k: v.to(device, non_blocking=True) for k, v in next(batches).items()
}
last = micro == ACCUM_STEPS - 1
# Skip the gradient all-reduce on all but the last micro-batch.
sync = contextlib.nullcontext() if last else model.no_sync()
with sync:
with torch.autocast("cuda", dtype=torch.bfloat16):
out = model(**batch, use_cache=False)
loss = out.loss / ACCUM_STEPS
loss.backward()
loss_sum += loss.detach()
tokens += batch["attention_mask"].sum()
grad_norm = torch.nn.utils.clip_grad_norm_(trainable, MAX_GRAD_NORM)
optimizer.step()
scheduler.step()
optimizer.zero_grad(set_to_none=True)
if step % LOG_EVERY == 0:
# Logging only: training already synced gradients, not losses.
stats = torch.stack([loss_sum, tokens])
dist.all_reduce(stats, op=dist.ReduceOp.SUM)
loss_avg = stats[0].item() / world_size
total_tokens = stats[1].item()
dt = time.perf_counter() - t0 # .item() above waited for the GPU
if is_main:
print(
f"epoch={epoch} step={step} loss={loss_avg:.4f} "
f"grad_norm={grad_norm.item():.3f} "
f"lr={scheduler.get_last_lr()[0]:.2e} "
f"tok/s={total_tokens / dt:,.0f}"
)
if is_main:
ckpt = os.path.join(OUTPUT_DIR, f"epoch-{epoch}")
model.module.save_pretrained(ckpt) # adapter weights only
torch.save(
{
"optimizer": optimizer.state_dict(),
"scheduler": scheduler.state_dict(),
"epoch": epoch,
},
os.path.join(ckpt, "trainer_state.pt"),
)
dist.barrier() # nobody races ahead while rank 0 writes
dist.destroy_process_group()
if __name__ == "__main__":
main()
Why each piece is there
Process group and device binding. Calling torch.cuda.set_device(local_rank) before touching any CUDA tensor ensures each process allocates memory strictly on its assigned GPU, and passing device_id=device into dist.init_process_group binds NCCL to that device immediately. LOCAL_RANK indexes the GPU within the current node (0..7), whereas RANK (via dist.get_rank()) is the global process ID across all nodes used to gate rank-0 logging and checkpointing.
Model loading and cache warm-up. Passing device_map={"": local_rank} streams the checkpoint directly onto each rank’s GPU. Make sure to pre-populate the Hugging Face cache once before launching torchrun (hf download google/gemma-4-12B-it), or all eight processes will race to download and unpack the same safetensors shards simultaneously. In current Transformers releases, AutoModelForCausalLM resolves the 12B checkpoint to its multimodal class (AutoModelForMultimodalLM works identically for text-only batches).
Scoped LoRA targets. LORA_TARGETS is a regex that matches only projection layers inside language_model (PEFT treats string target_modules as regular expressions). On a text-only dataset, attaching LoRA to vision or audio projections leaves those adapter parameters with None gradients after backward, which causes DDP to fault. Scoping the regex to language_model is much cleaner and faster than enabling find_unused_parameters=True, which forces DDP to traverse the autograd graph on every step.
Non-reentrant gradient checkpointing. Always pass gradient_checkpointing_kwargs={"use_reentrant": False} when combining gradient checkpointing with PEFT and DDP; the legacy reentrant implementation has well-known edge cases with frozen base weights and DDP hooks.
Two-stage seeding. Seeding with torch.manual_seed(SEED) before get_peft_model guarantees identical LoRA initialization across ranks (reinforced by DDP’s constructor broadcast), while re-seeding with torch.manual_seed(SEED + rank) after wrapping gives each rank independent dropout masks during training.
gradient_as_bucket_view=True. This flag aliases each parameter’s .grad tensor directly into the underlying DDP communication bucket instead of allocating a second buffer and copying between them, which saves 0.26 GB of duplicate memory and one copy per step.
Deterministic step counts. Setting drop_last=True on both DistributedSampler and DataLoader guarantees that len(loader) (and therefore steps_per_epoch = len(loader) // ACCUM_STEPS) is identical on every rank, so a ragged tail batch cannot hang the final all-reduce.
Accumulation with no_sync(). The first ACCUM_STEPS - 1 micro-batches run inside model.no_sync(), accumulating gradients locally in .grad while dividing each micro-batch loss by ACCUM_STEPS. The final micro-batch runs under nullcontext() so backward triggers the bucketed all-reduce across ranks.
bf16 autocast without GradScaler. The frozen base weights sit in bf16 while PEFT keeps the trainable LoRA weights in fp32 for numerical stability. torch.autocast("cuda", dtype=torch.bfloat16) runs the forward/backward matmuls in bf16 and accumulates into the fp32 adapter gradients. Because bfloat16 shares the exact 8-bit exponent range of float32, no GradScaler is needed.
Gradient clipping. clip_grad_norm_ runs after the final micro-batch’s backward pass, when gradients have already been averaged across all ranks, so every rank computes the exact same global gradient norm and scales its step identically.
Step logging. Every rank participates in the dist.all_reduce(stats) call on logging steps, and .item() forces a host-device synchronization before reading time.perf_counter() so tok/s reflects GPU execution time rather than asynchronous kernel launch latency.
Rank-0 checkpointing with a barrier. Because all replicas hold identical weights and optimizer state, only rank 0 writes model.module.save_pretrained(ckpt) (~262 MB for the fp32 adapter) and trainer_state.pt. The trailing dist.barrier() prevents ranks 1–7 from racing ahead into the next epoch while rank 0 flushes to disk.
Clean teardown. Calling dist.destroy_process_group() shuts down NCCL communicators cleanly before process exit.
What I’d add before a real run
- Assistant-only loss masking: The minimal script computes loss over the entire rendered conversation, user prompts included. For production instruction/chat tuning, mask prompt and system tokens to
-100inlabelsso gradients come strictly from assistant turns. - Checkpoint resume: Load the saved adapter back via PEFT, restore
optimizerandschedulerstate dicts fromtrainer_state.pt, and advance the epoch counter andDistributedSampler. - Distributed evaluation: Never call
model(...)on the DDP-wrapped model from rank 0 alone, as it will hang waiting for ranks 1–7. Either run validation across all ranks with aDistributedSamplerandall_reducethe validation loss, or run validation on rank 0 against the unwrappedmodel.moduleinsidetorch.no_grad()while ranks 1–7 wait at adist.barrier(). - Global token-weighted loss: Worth adding if sequence lengths vary widely across micro-batches (see batch size and learning rate).
Launch with torchrun
On a single node with eight GPUs:
pip install torch "transformers>=5.10.1" "peft>=0.19.0" accelerate
hf download google/gemma-4-12B-it
torchrun --standalone --nproc_per_node=8 train_ddp.py
--standalone spins up a local localhost rendezvous automatically. To scale the exact same script across two 8-GPU nodes, point both nodes at a shared rendezvous endpoint:
torchrun \
--nnodes=2 \
--nproc_per_node=8 \
--rdzv_backend=c10d \
--rdzv_endpoint=node0.example.internal:29400 \
--rdzv_id=gemma4-lora-001 \
train_ddp.py
Zero lines of Python change: WORLD_SIZE becomes 16, LOCAL_RANK still runs 0..7 on each machine, and rank 0 writes checkpoints to OUTPUT_DIR. See the torchrun elastic documentation for fault-tolerant flags like --max_restarts.
Across nodes, gradient reductions step off intra-node NVLink and traverse the network. For our ~0.26 GB LoRA gradient buffer, even standard datacenter Ethernet handles the all-reduce comfortably; for full-parameter fine-tuning (42 GB transferred per GPU per step), multi-node scaling is the reason teams buy InfiniBand or RoCE fabrics.
Run it on Modal
The GPU benchmarks in my Megatron visualizer run on Modal, where requesting a multi-GPU container is a one-line decorator (gpu="H100:8"). Following Modal’s recommendation to invoke torchrun as a subprocess inside the container function:
# modal_train.py
import os
import subprocess
import modal
app = modal.App("gemma4-ddp")
volume = modal.Volume.from_name("gemma4-ddp", create_if_missing=True)
image = (
modal.Image.debian_slim(python_version="3.12")
.pip_install("torch", "transformers>=5.10.1", "peft>=0.19.0", "accelerate")
.env({"HF_HOME": "/vol/hf"})
.add_local_file("train_ddp.py", "/root/train_ddp.py")
.add_local_file("train.jsonl", "/root/train.jsonl")
)
@app.function(image=image, gpu="H100:8", volumes={"/vol": volume}, timeout=6 * 60 * 60)
def train():
env = {
**os.environ,
"DATA_PATH": "/root/train.jsonl",
"OUTPUT_DIR": "/vol/checkpoints",
}
subprocess.run(
["torchrun", "--standalone", "--nproc_per_node=8", "/root/train_ddp.py"],
check=True,
env=env,
)
volume.commit()
@app.local_entrypoint()
def main():
train.remote()
Run it with modal run modal_train.py. Mounting the persistent Modal Volume at /vol caches the downloaded Hugging Face weights across runs and stores the output adapter checkpoints.
Failure modes that hang or corrupt a DDP job
Most distributed bugs don’t raise a clean Python traceback; they hang until the NCCL watchdog kills the job 10 to 30 minutes later because one rank entered a collective that another rank never reached. One invariant governs every failure mode below: every rank must issue the exact same collectives, in the exact same order, with the exact same tensor shapes.
Unequal numbers of batches
If rank 3 draws one more batch than rank 5, rank 3 enters a backward all-reduce while rank 5 moves on to the end-of-epoch barrier. Common culprits include:
- Custom samplers that don’t pad or trim evenly across
world_size. - Sharded
IterableDatasetstreams where underlying JSONL/Parquet shards have unequal row counts. - Filtering out bad or empty samples inside the training loop on a single rank (
if len(ids) == 0: continue).
Fix:
- Use
DistributedSampler(..., drop_last=True)andDataLoader(..., drop_last=True). - Derive
steps_per_epochfromlen(loader)so every rank runs the exact same loop bound. - Never
continuepast a micro-batch on one rank without executing the forward/backward pass on all ranks.
If uneven shard lengths are unavoidable (for example, streaming variable-length shards), wrap your training loop in PyTorch’s Join context manager so exhausted ranks continue participating in collectives as zero-gradient shadow processes until the slowest rank finishes.
Parameters that don’t receive gradients
Symptom: Step 0 succeeds, and step 1 crashes with RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one, listing parameter indices that never received a gradient.
Cause: A parameter with requires_grad=True wasn’t touched during the forward pass. On Gemma 4 checkpoints, the classic trigger is passing target_modules="all-linear" to PEFT on a multimodal model class, which attaches trainable LoRA matrices to vision and audio projection layers that a text-only batch never executes.
I reproduced this directly against a miniature test model built from the Gemma 4 12B config. With target_modules="all-linear", step 0 completes, and step 1 immediately raises Expected to have finished reduction in the prior iteration naming six unused multimodal projection parameters.
Fix: Scope target_modules to the subtree your data actually exercises (as LORA_TARGETS does above), or explicitly freeze unused towers before wrapping in DDP. Avoid slapping find_unused_parameters=True onto DDP(...) unless your active parameter set changes from batch to batch (such as unshared expert LoRA layers in an MoE model where sparse routing can skip an expert entirely).
Rank-divergent code paths
Calling dist.all_reduce(...) inside if rank == 0:, validating through the DDP wrapper on rank 0 alone, or calling break when a single rank hits a NaN loss will immediately desynchronize the NCCL ring.
Fix: Keep all collectives unconditional. When a control-flow decision like early stopping or NaN recovery depends on runtime state:
- Compute or aggregate the condition on rank 0.
dist.broadcastthe decision flag as a small tensor to all ranks.- Branch on the broadcast tensor so every rank takes the exact same exit path.
Model changes after the DDP wrap
DDP registers its autograd hooks and builds parameter buckets inside DDP.__init__. If you attach PEFT adapters, enable gradient checkpointing, or unfreeze layers after calling DDP(model), those parameters stay outside the communication buckets. Always finish mutating the model structure before wrapping it in DDP.
Doubled or missing synchronization with accumulation
If your forward pass runs outside with model.no_sync(): and only loss.backward() sits inside it, DDP ignores no_sync() and runs a full all-reduce on every micro-batch, which keeps the math right at a fraction of the throughput. Conversely, if you leave the last micro-batch inside no_sync(), DDP skips the all-reduce entirely and your replicas diverge over time. A quick sanity check when writing custom loops is to all_reduce a checksum of a trainable LoRA tensor after 10 steps and assert all ranks match.
DataLoader workers and pickling
Under the spawn or forkserver multiprocessing start methods (default on macOS and frequently used in containerized Linux/NCCL environments to avoid CUDA fork hazards), every object passed to a DataLoader worker must be picklable. My first draft of train_ddp.py used a nested closure for collate_fn and crashed immediately under spawn with AttributeError: Can't get local object 'make_collate.<locals>.collate', which is why PadCollate is defined as a top-level class.
Every rank writes checkpoints
Letting all eight ranks write to the same checkpoint directory on a shared filesystem (NFS, FSx, or a Modal Volume) produces truncated or corrupted safetensors files. Gate all disk writes behind if is_main: and follow them with dist.barrier().
NCCL and debugging environment variables
Whenever a multi-GPU job hangs or crawls, these are the first environment variables I reach for (full details in the NVIDIA NCCL environment variable reference and PyTorch distributed docs):
| Variable | What it does |
|---|---|
NCCL_DEBUG=INFO |
Logs NCCL initialization, including which transports (NVLink, PCIe, InfiniBand, sockets) it picked and the topology. |
NCCL_DEBUG_SUBSYS=INIT,NET |
Narrows the log to initialization and networking. |
NCCL_SOCKET_IFNAME=eth0 |
Pins the network interface. Multi-node jobs often hang because NCCL picked a Docker bridge or the wrong NIC. |
NCCL_IB_DISABLE=1 |
Disables InfiniBand, to test whether a hang is IB-specific. A diagnostic step, not a fix. |
TORCH_DISTRIBUTED_DEBUG=DETAIL |
Makes PyTorch check collective consistency and report which parameters didn’t receive gradients. |
TORCH_NCCL_TRACE_BUFFER_SIZE=2000 |
Enables PyTorch’s NCCL flight recorder, which records recent collectives per rank so you can see who stopped where. |
Two more habits save hours of debugging. Run nvidia-smi topo -m once on any new instance type so you know whether GPU pairs communicate over NVLink or traverse PCIe/SYS, and pass a short timeout=datetime.timedelta(minutes=2) to dist.init_process_group while debugging so collective hangs dump a stack trace in two minutes instead of thirty.
Measure throughput and scaling
The metric that matters is non-padding tokens processed per second across all ranks. To measure true scaling efficiency:
- Run the exact same configuration on 1, 2, 4, and 8 GPUs while holding per-GPU
MICRO_BATCHconstant. - Discard the first 5–10 steps so CUDA context initialization, caching allocator warmup, and NCCL ring setup don’t skew the numbers.
- Average
tok/sover a steady window of steps. - Compute scaling efficiency as
tokens_per_s(N) / (N × tokens_per_s(1)).
My Megatron visualizer’s live 2×T4 benchmark (api_multi_gpu_benchmark in app.py) computes this exact ratio (parallel_efficiency = speedup / ideal_speedup × 100) for a column-parallel matmul across two GPUs, separating raw compute time from GPU-to-GPU transfer overhead.
For LoRA fine-tuning under DDP, here is what you should expect to see:
- The all-reduce is tiny and mostly hidden. Synchronizing ~0.26 GB of
fp32LoRA gradients once per optimizer step, overlapped with backward computation over NVLink, takes only a few milliseconds, so single-node scaling efficiency across 8 GPUs typically lands well above 90%. - Gradient accumulation amortizes communication even further. With
ACCUM_STEPS=8, seven out of every eight backward passes run insideno_sync()with zero network traffic. - Your real bottleneck shifts to tokenization and padding waste. Once communication is nearly free, straggler ranks caused by mixing a 128-token example and a 2,048-token example in the same step dominate wall-clock time. Length-bucketing or sequence packing usually buys a much larger speedup than tweaking NCCL flags.
- Multi-node scaling depends on your fabric. Moving 0.46 GB per GPU per step is barely noticeable on 100+ Gbps networking, but on slow 10 Gbps Ethernet it will surface as visible gaps between steps.
To verify that backward and communication are overlapping as intended, capture a 3-step trace with torch.profiler and open it in Perfetto (ui.perfetto.dev). The ncclDevKernel_AllReduce stream should run concurrently alongside your backward GEMM kernels rather than sitting in a serialized block at the tail of the step. If the all-reduce waits until the very end, tuning bucket_cap_mb downward helps earlier buckets fire sooner.
If you want to build intuition for how data, tensor, and pipeline parallelism schedule micro-batches differently, play with the interactive data-parallel tab in the Megatron visualizer. And for a look at how multi-GPU training and inference fit into a broader Kubernetes CI/CD workflow, see the multi-GPU Kubernetes delivery platform case study.
Related work
Production ML context
See how this topic connects to production ML systems, infrastructure, and inference.