BigDansing: A System for Big Data Cleansing

著者: Zuhair Khayyat, Ihab F. Ilyas, Alekh Jindal, Samuel Madden, Mourad Ouzzani, Paolo Papotti, Jorge-Arnulfo Quiané-Ruiz, Nan Tang, Si Yin - ACM SIGMOD International Conference on Management of Data 2015 被引用: 151

Data cleansing approaches have usually focused on detect-ing and fixing errors with little attention to scaling to big datasets. This presents a serious impediment since data cleansing often involves costly computations such as enu-merating pairs of tuples, handling inequality joins, and deal-ing with user-defined functions. In this paper, we present BigDansing, a Big Data Cleansing system to tackle ef-ficiency, scalability, and ease-of-use issues in data cleans-ing. The system can run on top of most common general purpose data processing platforms, ranging from DBMSs to MapReduce-like frameworks. A user-friendly program-ming interface allows users to express data quality rules both declaratively and procedurally, with no requirement of being aware of the underlying distributed platform. BigDansing takes these rules into a series of transformations that enable distributed computations and several optimizations, such as shared scans and specialized joins operators. Experimental results on both synthetic and real datasets show that Big-Dansing outperforms existing baseline systems up to more than two orders of magnitude without sacrificing the quality provided by the repair algorithms. 1.