Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Gradient Decomposition Methods for Training Neural Networks with Non-Ideal Synaptic Devices

Published

Author(s)

Junyun Zhao, Siyuan Huang, Osama Yousuf, Yutong Gao, Brian Hoskins, Gina Adam

Abstract

While promising for high capacity machine learning accelerators, memristor devices have non-idealities that prevent software-equivalent accuracies when used for online training. This work uses a combination of Mini-Batch Gradient Descent (MBGD) to average out gradients, stochastic rounding to avoid vanishing weight updates, and decomposition methods to keep the memory overhead low during mini-batch training. Since the weight update has to be transferred to the memristor matrices efficiently, we also investigate the impact of reconstructing the gradient matrixes both internally (rank-seq) and externally (rank-sum) to the memristor array. Our results show that streaming batch principal component analysis (streaming batch PCA) and non-negative matrix factorization (NMF) decomposition algorithms can achieve near MBGD accuracy in a memristor-based multi-layer perceptron trained on MNIST with only 3 ranks at significant memory savings. Moreover, NMF rank-seq outperforms streaming batch PCA rank-seq at low-ranks making it more suitable for hardware implementations in memristor accelerators.
Citation
Frontiers in Neuroscience
Volume
15

Keywords

artificial intelligence, resistive switch, matrix decomposition

Citation

Zhao, J. , Huang, S. , Yousuf, O. , Gao, Y. , Hoskins, B. and Adam, G. (2021), Gradient Decomposition Methods for Training Neural Networks with Non-Ideal Synaptic Devices, Frontiers in Neuroscience, [online], https://doi.org/10.3389/fnins.2021.749811, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=932867 (Accessed May 29, 2024)

Issues

If you have any questions about this publication or are having problems accessing it, please contact reflib@nist.gov.

Created November 22, 2021, Updated November 29, 2022