Back to overview

Rock You like a Hurricane: Taming Skew in Large Scale Analytics

Type of publication Peer-reviewed
Publikationsform Proceedings (peer-reviewed)
Author Laurent Bindschaedler Jasmina Malicevic Nicolas Schiper Ashvin Goel and Willy Zwaenepoel,
Project Building Flexible Large-Graph Processing Systems on Commodity Hardware
Show all

Proceedings (peer-reviewed)

Title of proceedings Eurosys 2018

Open Access


Current cluster computing frameworks suffer from load imbalance and limited parallelism due to skewed data distributions, processing times, and machine speeds. We observe that the underlying cause for these issues in current systems is that they partition work statically. Hurricane is a high-performance large-scale data analytics system that successfully tames skew in novel ways. Hurricane performs adaptive work partitioning based on load observed by nodes at runtime. Overloaded nodes can spawn clones of their tasks at any point during their execution, with each clone processing a subset of the original data. This allows the system to adapt to load imbalance and dynamically adjust task parallelism to gracefully handle skew. We support this design by spreading data across all nodes and allowing nodes to retrieve data in a decentralized way. The result is that Hurricane automatically balances load across tasks, ensuring fast completion times. We evaluate Hurricane’s performance on typical analytics workloads and show that it significantly outperforms state- of-the-art systems for both uniform and skewed datasets, because it ensures good CPU and storage utilization in all cases.