Gradient Decomposition Methods for Training Neural Networks with Non-Ideal Synaptic Devices
Junyun Zhao, Siyuan Huang, Osama Yousuf, Yutong Gao, Brian Hoskins, Gina Adam
While promising for high capacity machine learning accelerators, memristor devices have non-idealities that prevent software-equivalent accuracies when used for online training. This work uses a combination of Mini-Batch Gradient Descent (MBGD) to average out gradients, stochastic rounding to avoid vanishing weight updates, and decomposition methods to keep the memory overhead low during mini-batch training. Since the weight update has to be transferred to the memristor matrices efficiently, we also investigate the impact of reconstructing the gradient matrixes both internally (rank-seq) and externally (rank-sum) to the memristor array. Our results show that streaming batch principal component analysis (streaming batch PCA) and non-negative matrix factorization (NMF) decomposition algorithms can achieve near MBGD accuracy in a memristor-based multi-layer perceptron trained on MNIST with only 3 ranks at significant memory savings. Moreover, NMF rank-seq outperforms streaming batch PCA rank-seq at low-ranks making it more suitable for hardware implementations in memristor accelerators.
, Huang, S.
, Yousuf, O.
, Gao, Y.
, Hoskins, B.
and Adam, G.
Gradient Decomposition Methods for Training Neural Networks with Non-Ideal Synaptic Devices, Frontiers in Neuroscience, [online], https://doi.org/10.3389/fnins.2021.749811, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=932867
(Accessed September 27, 2023)