PyTorch Brings Zephon Data Loading to TorchTitan

PyTorch has introduced Zephon into TorchTitan to solve the problem of data loader non-determinism during distributed model training. Standard data loaders alter sample order whenever GPU allocations change, causing loss curves to diverge upon resuming checkpoints. Testing on a 1B-parameter model checkpointed at step 150 on 16 GPUs and resumed on 8, 32, and 64 GPUs showed that traditional loaders diverge, whereas Zephon produces identical global batch SHA-256 hashes across all cluster sizes.

When training the 1B-parameter model from scratch across 20B tokens on 8, 16, 32, and 64 GPUs, standard data loaders caused score variations up to 0.82 points on DCLM Core v1 and 0.64 points on FineWeb. Zephon reduced this variance to 0.05 points on DCLM Core v1 and 0.011 points on FineWeb. By handling preprocessing on the fly—including tokenizing, packing, shuffling, and mixing—Zephon removes the need to store static preprocessed datasets while maintaining throughput comparable to Google Grain.

Under the hood, Zephon uses single-threaded accumulators to make deterministic decisions, separating stateful logic from parallel workers. It splits data streams into logical lanes rather than GPU ranks, redistributing these lanes when GPU counts change. Small checkpoints of a few MiB are maintained by using flush sentinels and replaying inputs rather than saving massive internal buffer states. To ensure deterministic execution, practitioners select fixed lane counts that divide evenly across data-parallel replicas, while configuring parameters like context length max length 2048 and parallelism workers.

For practitioners, this eliminates dataset storage overhead and prevents hardware allocation changes from corrupting ablation studies. Zephon also supports Python 3.14t, where a thread-based runner achieves a 3.7x speedup over standard process-based runners when fetching vision-language samples. The team is also developing dynamic curricula tools while expanding support alongside existing frameworks like Mosaic Streaming and Megatron-Energon.