https://www.zzudongxiang.com/tags/zero/Recent content in ZeRO on zzudongxiangHugozh-cn2026 zzudongxiang.com · 豫ICP备2022012018号-1 · 豫公网安备41160202000614号Mon, 01 Apr 2024 00:00:00 +0000https://www.zzudongxiang.com/posts/data-parallelism/Mon, 01 Apr 2024 00:00:00 +0000https://www.zzudongxiang.com/posts/data-parallelism/<div> <blockquote> <p>参考文献：</p> <p><span style="font-size: 14.4px">Team D, Majumder R, President V, et al. DeepSpeed: Extreme-scale model training for everyone</p> \[J\]<p>. Microsoft, 2020.</span></p> <p><span style="font-size: 14.4px">Rajbhandari S, Rasley J, Ruwase O, et al. ZeRO: Memory optimizations Toward Training Trillion Parameter Models</p> \[C\]<p>, 2020. IEEE, 2020-01-01.</span></p> <p>Ren J, Rajbhandari S, Aminabadi R Y, et al. ZeRO-Offload: Democratizing Billion-Scale Model Training</p> \[J\]<p>. <a href="http://arXiv.org" rel="noopener noreferrer nofollow" target="_blank">arXiv.org</a>, 2021.</p> <p><span style="font-size: 14.4px">Rajbhandari S, Ruwase O, Rasley J, et al. ZeRO-Infinity: Breaking the GPU MemoryWall for Extreme Scale Deep Learning: SC&rsquo; 21, November 14–19, 2021, St. Louis, MO, USA</p>