Back to overview

Failure Analysis and Prediction for Big-Data Systems

Type of publication Peer-reviewed
Publikationsform Original article (peer-reviewed)
Author Rosà Andrea, Chen Lydia Y., Binder Walter,
Project LoadOpt - Workload Characterization and Optimization for Multicore Systems
Show all

Original article (peer-reviewed)

Journal IEEE Transactions on Services Computing
Volume (Issue) 10(6)
Page(s) 984 - 998
Title of proceedings IEEE Transactions on Services Computing
DOI 10.1109/tsc.2016.2543718


Motivated by the high complexity of today’s datacenters, a large body of studies tries to understand workloads and resource utilization in datacenters. However, there is little work on exploring unsuccessful job and task executions. In this article, we study and predict three types of unsuccessful executions in traces of a Google datacenter, namely fail, kill, and eviction. We first quantitatively show their strongly negative impact on machine time and the resulting task slowdown. We analyze patterns of unsuccessful jobs and tasks, particularly focusing on their interdependencies, and we uncover their root causes by inspecting key workload and system attributes. Furthermore, we develop three on-line prediction models that can classify jobs and events into four classes upon arrival time, using independent or nested Neural Networks. We explore different combinations of feature sets and techniques to reduce the computational overhead. Our evaluation results show that the proposed models can accurately classify 94.4% of jobs and 76.8% of events into four classes.