Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Low-Rank Gradient Descent for Memory-Efficient Training of Deep In-Memory Arrays

Published

Author(s)

Siyuan Huang, Brian Hoskins, Matthew Daniels, Mark Stiles, Gina C. Adam

Abstract

The movement of large quantities of data during the training of a Deep Neural Network presents immense challenges for machine learning workloads. To minimize this overhead, espe- cially on the movement and calculation of gradient information, we introduce streaming batch principal component analysis as an update algorithm. Streaming batch principal component analysis uses stochastic power iterations to generate a stochastic k-rank approximation of the network gradient. We demonstrate that the low rank updates produced by streaming batch principal component analysis can effectively train convolutional neural networks on a variety of common datasets, with performance comparable to standard mini batch gradient descent. These results can lead to both improvements in the design of application specific integrated circuits for deep learning and in the speed of synchronization of machine learning models trained with data parallelism.
Citation
ACM Journal on Emerging Technologies in Computing Systems
Volume
19
Issue
2

Citation

Huang, S. , Hoskins, B. , Daniels, M. , Stiles, M. and Adam, G. (2023), Low-Rank Gradient Descent for Memory-Efficient Training of Deep In-Memory Arrays, ACM Journal on Emerging Technologies in Computing Systems, [online], https://doi.org/10.1145/3577214 (Accessed May 24, 2024)

Issues

If you have any questions about this publication or are having problems accessing it, please contact reflib@nist.gov.

Created May 18, 2023, Updated September 22, 2023