Publication
An Architecture for Compiling UDF-centric Workflows
Andrew Crotty; Alex Galakatos; Kayhan Dursun; Tim Kraska; Carsten Binnig; Ugur Çetintemel; Stan Zdonik
In: Proceedings of the VLDB Endowment (PVLDB), Vol. 8, No. 12, Pages 1466-1477, Association for Computing Machinery (ACM), 2015.
Abstract
Data analytics has recently grown to include increasingly sophisticated techniques, such as machine learning and advanced statistics. Users frequently express these complex analytics tasks as workflows of user-defined functions (UDFs) that specify each algorithmic step. However, given typical hardware configurations and dataset sizes, the core challenge of complex analytics is no longer sheer data volume but rather the computation itself, and the next generation of analytics frameworks must focus on optimizing for this computation bottleneck. While query compilation has gained widespread popularity as a way to tackle the computation bottleneck for traditional SQL workloads, relatively little work addresses UDF-centric workflows in the domain of complex analytics.