Back to overview

PCA-Initialized Deep Neural Networks Applied to Document Image Analysis

Type of publication Peer-reviewed
Publikationsform Original article (peer-reviewed)
Author Seuret Mathias, Alberti Michele, Liwicki Marcus, Ingold Rolf,
Project HisDoc III : Large-Scale Historical Document Classification
Show all

Original article (peer-reviewed)

Journal 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR)
Volume (Issue) 01
Page(s) 877 - 882
Title of proceedings 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR)
DOI 10.1109/icdar.2017.148


In this paper, we present a novel approach for initializing deep neural networks, i.e., by using Principal Component Analysis (PCA) to initialize neural layers. Usually, the initialization of the weights of a deep neural network is done in one of the three following ways: 1) with random values, 2) layer-wise, usually as Deep Belief Network or as auto-encoder, and 3) re-use of layers from another network (transfer learning). Therefore, typically, many training epochs are needed before meaningful weights are learned, or a rather similar dataset is required for seeding a fine-tuning of transfer learning. In this paper, we describe how to turn a PCA into an auto-encoder, by generating an encoder layer of the PCA parameters and furthermore adding a decoding layer. We analyze the initialization technique on real documents. First, we show that a PCA-based initialization is quick and leads to a very stable initialization. Furthermore, for the task of layout analysis we investigate the effectiveness of PCA-based initialization and show that it outperforms state-of-the-art random weight initialization methods.