One-sentence Summary

Google proposes the Dataflow Model, a set of principled abstractions for processing unbounded, unordered, out-of-order datasets that enables practitioners to explicitly choose tradeoffs along correctness, latency, and cost rather than forcing data into finite, complete pools, validated through real-world production experience.

Key Contributions

• The paper introduces the Dataflow Model, a unified model for unbounded datasets that lets practitioners make explicit tradeoffs among correctness, latency, and cost through principled abstractions such as event-time ordering, unaligned windows, flexible triggering, integrated accumulation, and retraction.
• The paper defines the model's core semantics and design principles by separating what results are computed, where they are computed in event time, when they are materialized in processing time, and how earlier results relate to later refinements, while abstracting away differences among batch, micro-batch, and streaming engines.
• The paper supports the model with analysis and real-world Google-scale unbounded data processing experiences, including MillWheel and large log joining pipelines, showing how the model addresses event-time skew, weakly consistent streaming results, and unified streaming and batch implementations.

Introduction

Modern data processing systems must handle massive, unbounded, and frequently out of order data streams while producing timely, correct results for use cases such as advertising analytics, operations monitoring, and near real time decision making. The authors explain that existing batch systems can be too slow because they process collected inputs, while many streaming systems struggle with fault tolerance, exactly once correctness, expressive event time windowing, session windows, or simplicity at global scale. To address these gaps, the authors leverage a unified programming model that decouples the logical pipeline from the underlying batch, micro batch, or streaming engine. The model supports unaligned event time windows, flexible triggering, incremental updates and retractions, and explicit control over the tradeoff among correctness, latency, and cost.

Method

The authors leverage a unified dataflow model that describes batch, micro batch, and streaming pipelines with a single set of semantics. In this model, pipeline data are conceptually treated as tuples of the form $(key, value, event time, window)$(key,value,eventtime,window). The event time timestamp can be assigned at input or modified during processing, while the window field determines where in event time a datum is grouped. This representation allows the same pipeline definition to express pure batch computations, low latency stream processing, and hybrid patterns similar to Lambda Architecture pipelines.

At the core of the framework are two parallel primitives. The first is ParDo, which applies a user defined DoFn to each input element. A DoFn may emit zero or more output elements for each input, so ParDo supports map, filter, expand, and other element wise transformations. The second primitive is GroupByKey, which collects all values associated with the same key. GroupByKey is powerful, but it becomes problematic on unbounded streams because the system cannot know when the last value for a key has arrived. The model therefore couples grouping with time bounded windows, so that aggregation is performed over a key and a window rather than over a key alone.

The authors make windowing a first class part of the model by separating it into two operations. AssignWindows places each input element into zero or more windows based on its timestamp. For example, an element assigned to sliding windows may be duplicated into each window that contains its timestamp. MergeWindows is applied during grouping and allows overlapping or adjacent windows to be combined. This separation is essential for data driven windowing schemes such as sessions. In session windowing, each element initially creates a provisional window extending beyond its event time, and later grouping can merge overlapping session windows into a single session.

To make this concrete, the authors consider a summation pipeline where GroupByKey becomes a composite GroupByKeyAndWindow operation. This operation conceptually consists of DropTimestamps, GroupByKey, MergeWindows, GroupAlsoByWindow, and ExpandToElements. The DropTimestamps step is useful because window membership becomes the primary grouping signal after assignment. MergeWindows implements strategy specific merging, such as combining overlapping sessions. GroupAlsoByWindow produces per key, per window groups of values, and ExpandToElements materializes grouped results back into timestamped elements. The authors design the model so that all windows are treated as unaligned from the model perspective, while execution engines may still apply optimizations when aligned window patterns are detected.

The example input data are useful for understanding how the model behaves over time. The data contain ten integer observations for a single key. The horizontal axis represents event time, meaning when the event actually occurred, while the vertical axis represents processing time, meaning when the pipeline observes the event. The visualization also shows two watermark paths: an ideal watermark that advances in perfect lockstep with event time, and a realistic watermark that can be delayed or inaccurate because of distributed systems skew. One observation, with value 9, arrives behind the watermark and is therefore late. This motivates the need for mechanisms that do not assume perfect completeness.

When this pipeline is run in classic batch mode, the behavior is straightforward. The system waits until the bounded input is complete, groups all values for the key into one bundle, and emits a single sum. Because classic batch processing is usually event time agnostic, the default window is a global window covering all event time, and the result covers the full processing time of the job. The shaded region in the next visualization represents the total sum of 51, materialized after all inputs have been processed.

For unbounded streaming sources, the same global window cannot rely on the default watermark trigger alone, because a global window never becomes complete. The authors solve this by introducing triggers, which determine when in processing time the current result for a window should be emitted as a pane. A pane is not necessarily the final answer for a window; it can be an estimate, a partial result, or a refined result. This trigger mechanism is complementary to windowing. Windowing chooses where data are grouped in event time, while triggering chooses when grouped results are observed in processing time.

If the goal is to maintain a global per key sum and receive periodic updates, the authors can apply a repeating periodic trigger and use accumulating mode. In accumulating mode, each new pane is a refinement of the previous pane, because the window contents remain available and newer results overwrite or update older results downstream. The visualization of this execution shows updated global sums at one minute processing time boundaries. The overlapping output regions reflect that each emission includes data accumulated from prior processing time intervals.

Changing the accumulation mode to discarding changes the output semantics while preserving the trigger structure. In discarding mode, each trigger fire emits the result for the current pane and then clears the window state. Later panes are therefore independent of earlier panes, and the output represents deltas rather than cumulative totals. This pattern closely matches processing time windowing behavior found in many streaming systems, even though it is implemented through the unified trigger and windowing model. The visualization shows non overlapping output regions whose sums correspond to data observed between trigger fires.

The model can also express tuple based windows by using data driven triggers. Instead of triggering based on time, a trigger can fire when a certain count of elements has been observed. When the authors set the trigger to fire after every two records and use discarding mode, each pane contains the sum of two adjacent observations by processing time. This produces tuple window semantics without introducing a separate windowing abstraction. The final visualization illustrates this execution, where output panes are generated in response to data arrival counts rather than periodic clocks or event time boundaries.

Beyond time based and count based triggers, the authors support watermark triggers, percentile watermark triggers, processing time triggers, and custom external signals. Triggers can also be composed using logical operations, loops, and sequences. This composability allows a pipeline to emit an early partial result when data arrive, then later emit a more complete result when the watermark advances, and optionally emit an updated result when late data arrive. The refinement relationship among multiple panes is controlled by discarding, accumulating, and accumulating with retracting modes. In accumulating and retracting mode, each update is accompanied by a retraction of the previous value, which is important when a window produces multiple results that are later grouped again downstream. This design lets the pipeline adapt to both early approximate results and delayed corrections without relying on a separate batch recovery pathway.