Back to overview

Information Bottleneck Classification in Extremely Distributed Systems

Type of publication Peer-reviewed
Publikationsform Original article (peer-reviewed)
Author Ullmann Denis, Rezaeifar Shideh, Taran Olga, Holotyak Taras, Panos Brandon, Voloshynovskiy Slava,
Project Machine Learning based Analytics for Big Data in Astronomy
Show all

Original article (peer-reviewed)

Journal Entropy
Volume (Issue) 22(11)
Page(s) 1237 - 1237
Title of proceedings Entropy
DOI 10.3390/e22111237

Open Access

Type of Open Access Publisher (Gold Open Access)


We present a new decentralized classification system based on a distributed architecture. This system consists of distributed nodes, each possessing their own datasets and computing modules, along with a centralized server, which provides probes to classification and aggregates the responses of nodes for a final decision. Each node, with access to its own training dataset of a given class, is trained based on an auto-encoder system consisting of a fixed data-independent encoder, a pre-trained quantizer and a class-dependent decoder. Hence, these auto-encoders are highly dependent on the class probability distribution for which the reconstruction distortion is minimized. Alternatively, when an encoding–quantizing–decoding node observes data from different distributions, unseen at training, there is a mismatch, and such a decoding is not optimal, leading to a significant increase of the reconstruction distortion. The final classification is performed at the centralized classifier that votes for the class with the minimum reconstruction distortion. In addition to the system applicability for applications facing big-data communication problems and or requiring private classification, the above distributed scheme creates a theoretical bridge to the information bottleneck principle. The proposed system demonstrates a very promising performance on basic datasets such as MNIST and FasionMNIST.