Data By the Bay

This talk traces the evolution of ideas in Google's data processing tools over the past 13 years - from classic MapReduce, to strongly consistent stream processing with Millwheel, to the unified batch and streaming programming model of Apache Beam.

Originally based on years of experience developing Big Data infrastructure within Google (such as MapReduce, FlumeJava, and MillWheel), it has now been donated to the OSS community at large.

Beam cleanly separates the different aspects of temporal data processing: what computation to apply, where in event time to apply it, when in processing time to produce results, and how to refine the results as late data arrives. By decoupling semantics from the underlying execution environment, Beam provides portability across multiple runners, both proprietary (e.g., Google Cloud Dataflow) and open-source (e.g., Flink, Spark, et al).

I will give an overview of the programming model and current status of the project and invite you to participate in its rapidly developing ecosystem.