Frequently Asked Questions
1. Source / Citation
Based on Appendix A.2 and A.3 of "Communication-Efficient Language Model Training Scales Reliably and Robustly".
Model assumes bfloat16 precision.
2. Wouldn't a real-world implementation of Streaming Diloco need some form of Hierarchical All-Reduce?
I think so, most likely. Real-world implementations typically use hierarchical collectives.
However, the Scaling Laws paper explicitly models the time using the standard flat ring lower bound ($T \approx 2N/W$). This implies that if the Inter-DC link is the bottleneck, the total transfer time is governed by that bottleneck bandwidth ($W$) regardless of the reduction topology.
3. What are Staleness (S) and Fragments (P)?
Fragments (P): Streaming DiLoCo breaks the outer sync payload into $P$ chunks. While this doesn't change the total bytes sent, it allows for finer-grained overlap and lower memory overhead for buffers.
Staleness (S): By allowing the outer optimizer to be $S$ steps stale, the communication of gradients can be spread over $S$ intervals of length $H$. This effectively reduces the bandwidth requirement by a factor of $S$ (or allows using a link $S$ times slower without bottlenecking).
4. Why is "Bandwidth (Per Chip)" used in the formulas?
The training time calculation uses the bandwidth-optimal All-Reduce cost model from Patarasuk & Yuan (2009), approximated in the paper as T ≈ 2N / W.
In this formula, W represents the per-chip link bandwidth.
However, the "Inter-DC Capacity" metric displayed in the results represents the aggregate cross-sectional bandwidth that would be required if every chip in the datacenter saturated its Inter-DC link simultaneously. It is calculated as Bandwidth_Per_Chip × (R / M).
5. Why do Vanilla and Streaming DiLoCo lines overlap?
In the simulator's model, Streaming DiLoCo sends the same total amount of data as Vanilla DiLoCo, just in smaller chunks.
The only difference is the added latency overhead of sending multiple fragments ($P \times \epsilon$).
For large models (e.g., 100B params), the bandwidth transfer time (tens of seconds) massively dominates the latency overhead (milliseconds), making the lines appear identical on the plot.
To see them diverge, try increasing the Fragments count or the Inter-DC Latency.