Why State flows with Data instead of being Centralized by the controller? #7429

That is the consequence of having the state flowing with the data, but what is the reason to avoid global state?

The reason we want to avoid global state in a pipelined distributed workflow system is probably similar to why we avoid global variables in a multithreaded program. We don't want one thread to modify a variable while another thread is using it.

The same principle applies to the workflow. We don't want a downstream operator to modify some shared state while an upstream operator is still using it. We had a similar issue with large binaries. You can look at the previous discussions to see why we need locking for large binaries. The same concept is also covered in CS223, Transaction Processing and Distributed Data Management.

I’m not sure the global-variable analogy applies to the current design.

As I understand it, once state is emitted into the workflow, operators do not share and mutate the same state object. Instead, each operator receives a state message and may forward a transformed version of it. That avoids concurrent mutation, but it does not eliminate state-management problems. It shifts them into message ordering, duplicated versions, stale reads, and deciding which transformed state is authoritative.

A centralized design would not necessarily mean exposing one mutable object to every operator. The controller could maintain an immutable or versioned snapshot, allow operators to read it without locking, and serialize only the relatively rare updates. If most operators only read the state, lock contention should be limited. If, as I currently understand, only operators such as Loop End modify it, then the write path may be narrow enough to manage explicitly.

In fact, centralizing writes could provide stronger consistency because there would be one place responsible for validating updates, ordering them, and publishing the latest version. By contrast, when state travels through the dataflow, multiple transformed copies can exist simultaneously, and correctness depends on the workflow topology and message ordering.

So I think the key question is not simply "global state versus no global state." It is:

What concrete property does carrying state through the dataflow provide that a centralized, versioned, read-mostly state service would not?

For example, is the goal to improve replayability, fault recovery, scalability, or operator isolation? Those would be stronger reasons than the shared-variable analogy alone.

Please correct me if other operators currently modify the state, but my understanding is that Loop End is the main transformation point.

I did not know about this 🫡

The paper should answer most of the questions you have. State is implemented using the ECM markers discussed in the paper.

Thanks for the discussion. @carloea2: If you think the paper answered your question, it will be good to summarize the reason. It's also beneficial to include 1-2 related architecture diagrams (from the paper) in this discussion to make the context more clear.

For our record, the paper is the following:

Responsive State Retrieval in Pipelined Executions of Dataflows, by Shengquan Ni and Chen Li, HILDA workshop at SIGMOD 2025

Well actually it did not answer it, but I think is fine, I got more understanding out of the paper, thanks.