NIST Researchers Demonstrate Improved Way of Training Distributed Digital Devices

Distributed digital devices – such as smart phones and more – are updated and trained by means of “Federated Learning.” Basically, it involves a central server sharing a training model with selected devices. Each device updates the model with its local data and then sends it back to the central server, which converges devices’ inputs into a new model. The central server repeats the process until update criteria are met.

Federated Learning enables collaborative learning of a shared model without exposing devices’ local data. The problem with Federated Learning is that the convergence process can be slow because of data differences. The devices collect data from different sources, using different tools, under different conditions, and/or may have access only to partial or biased data, which can cause data distributions to differ among devices. Thus, the data is not what some call “independent and identically distributed” which is needed for fast Federated Learning.

NIST researchers offer an improved process in their paper Federated Learning with Server Learning for Non-IID Data, which was presented at the IEEE 57th Annual Conference on Information Sciences and Systems. Researchers refer to this improvement as “server learning.” The central server collects a small amount of training data, learns from it, and then incrementally distills the knowledge into a training model. This process enables the server to reconcile and converge differing data and doesn’t demand increased computing storage and communications for the devices. This process was aided by the researchers’ development of an algorithm and technical analysis.

Researchers evaluated the process’s accuracy and convergence rate using two datasets. The results were compared against conventional Federated Learning. Researchers found that the server learning process provided more accurate updates and training and needed only a small dataset to achieve meaningful performance gains.