usenix conference policies
HadoopProv: Towards Provenance as a First Class Citizen in MapReduce
We introduce HadoopProv, a modified version of Hadoopthat implements provenance capture and analysis in MapReduce jobs. It is designed to minimise provenancecapture overheads by (i) treating provenance tracking in Map and Reduce phases separately, and (ii) deferringconstruction of the provenance graph to the query stage. Provenance graphs are later joined on matching intermediate keys of the Map and Reduce provenance files. In our prototype implementation, HadoopProv has an overhead below 10% on typical job runtime (<7% and <30% average temporal increase on Map and Reduce tasks respectively). Additionally, we demonstrate that provenance queries are serviceable in O (k log n), where n is the number of records per Map task and k is the set of Map tasks in which the key appears.
Sherif Akoush, Ripduman Sohan, and Andy Hopper, University of Cambridge
Open Access Content
Papers are restricted to registered attendees until the event begins. Once the event begins, the content becomes free and open to everyone. Journal articles are open to everyone upon publication. If available, video, audio, and/or slides of this presentation will be posted here after the event.
We introduce HadoopProv, a modified version of Hadoopthat implements provenance capture and analysis in MapReduce jobs. It is designed to minimise provenancecapture overheads by (i) treating provenance tracking in Map and Reduce phases separately, and (ii) deferringconstruction of the provenance graph to the query stage. Provenance graphs are later joined on matching intermediate keys of the Map and Reduce provenance files. In our prototype implementation, HadoopProv has an overhead below 10% on typical job runtime (<7% and <30% average temporal increase on Map and Reduce tasks respectively). Additionally, we demonstrate that provenance queries are serviceable in O (k log n), where n is the number of records per Map task and k is the set of Map tasks in which the key appears.
connect with us