Maximizing Parallelism in Distributed Training for Huge Neural Networks

Bian, Zhengda; Xu, Qifan; Wang, Boxiang; You, Yang

doi:10.48550/arXiv.2105.14450

33 captures

24 Mar 2022 - 08 Mar 2026

Feb	MAR	Apr
	15
2022	2023	2024

success

fail

About this capture

COLLECTED BY

Collection: Save Page Now Outlinks

TIMESTAMPS

The Wayback Machine - http://web.archive.org/web/20230315075441/https://arxiv.org/abs/2105.14450

Global Survey

In just 3 minutes help us understand how you see arXiv.

We gratefully acknowledge support from the Simons Foundation and member institutions.

> cs> arXiv:2105.14450

quick links

●Login ●Help Pages ●About

Computer Science > Distributed, Parallel, and Cluster Computing

arXiv:2105.14450 (cs) [Submitted on 30 May 2021]

Title:Maximizing Parallelism in Distributed Training for Huge Neural Networks

Authors:Zhengda Bian, Qifan Xu, Boxiang Wang, Yang You Download PDF Abstract: The recent Natural Language Processing techniques have been refreshing the state-of-the-art performance at an incredible speed. Training huge language models is therefore an imperative demand in both industry and academy. However, huge language models impose challenges to both hardware and software. Graphical processing units (GPUs) are iterated frequently to meet the exploding demand, and a variety of ASICs like TPUs are spawned. However, there is still a tension between the fast growth of the extremely huge models and the fact that Moore's law is approaching the end. To this end, many model parallelism techniques are proposed to distribute the model parameters to multiple devices, so as to alleviate the tension on both memory and computation. Our work is the first to introduce a 3-dimensional model parallelism for expediting huge language models. By reaching a perfect load balance, our approach presents smaller memory and communication cost than existing state-of-the-art 1-D and 2-D model parallelism. Our experiments on 64 TACC's V100 GPUs show that our 3-D parallelism outperforms the 1-D and 2-D parallelism with 2.32x and 1.57x speedup, respectively.

Comments:	Technical Report of NUS HPC-AI Lab (this https URL). The leading two authors have equal contributions
Subjects:	Distributed, Parallel, and Cluster Computing (cs.DC); Machine Learning (cs.LG); Performance (cs.PF)
Cite as:	arXiv:2105.14450 [cs.DC]
	(or arXiv:2105.14450v1 [cs.DC] for this version)
	https://doi.org/10.48550/arXiv.2105.14450

Submission history

From: Yang You [view email] [v1] Sun, 30 May 2021 07:41:08 UTC (1,354 KB) Full-text links:

Download:

●PDF ●Other formats (license) Current browse context: cs.DC < prev | next > new | recent | 2105 Change to browse by: cs cs.LG cs.PF

References & Citations

●NASA ADS ●Google Scholar ●Semantic Scholar

DBLP - CS Bibliography

listing | bibtex Boxiang Wang Yang You export bibtex citation

Bibtex formatted citation

Data provided by:

Bookmark

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer (What is the Explorer?) Litmaps (What is Litmaps?) scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

Papers with Code (What is Papers with Code?) ScienceCast (What is ScienceCast?)

Demos

Replicate (What is Replicate?) Hugging Face Spaces (What is Spaces?) Related Papers

Recommenders and Search Tools

Connected Papers (What is Connected Papers?) CORE Recommender (What is CORE?) About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs. Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?) ●About ●Help ● ntact arXiv Contact ● ick here to subscribe Subscribe ●Copyright ●Privacy Policy ●Web Accessibility Assistance Get status notifications via emailorslack