Gradient Decomposition Methods for Training Neural Networks with Non-Ideal Synaptic Devices

Published

November 22, 2021

Author(s)

Junyun Zhao, Siyuan Huang, Osama Yousuf, Yutong Gao, Brian Hoskins, Gina Adam

Abstract

While promising for high capacity machine learning accelerators, memristor devices have non-idealities that prevent software-equivalent accuracies when used for online training. This work uses a combination of Mini-Batch Gradient Descent (MBGD) to average out gradients, stochastic rounding to avoid vanishing weight updates, and decomposition methods to keep the memory overhead low during mini-batch training. Since the weight update has to be transferred to the memristor matrices efficiently, we also investigate the impact of reconstructing the gradient matrixes both internally (rank-seq) and externally (rank-sum) to the memristor array. Our results show that streaming batch principal component analysis (streaming batch PCA) and non-negative matrix factorization (NMF) decomposition algorithms can achieve near MBGD accuracy in a memristor-based multi-layer perceptron trained on MNIST with only 3 ranks at significant memory savings. Moreover, NMF rank-seq outperforms streaming batch PCA rank-seq at low-ranks making it more suitable for hardware implementations in memristor accelerators.

Citation

Frontiers in Neuroscience

Volume

15

Pub Type

Journals

Download Paper

https://doi.org/10.3389/fnins.2021.749811

Local Download

Keywords

artificial intelligence, resistive switch, matrix decomposition

Citation

Zhao, J. , Huang, S. , Yousuf, O. , Gao, Y. , Hoskins, B. and Adam, G. (2021), Gradient Decomposition Methods for Training Neural Networks with Non-Ideal Synaptic Devices, Frontiers in Neuroscience, [online], https://doi.org/10.3389/fnins.2021.749811, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=932867 (Accessed October 10, 2025)