Designing a mechanism to synchronize states with commands despite timing / network inconsistencies

Question

I have two browser clients: A and B. They communicate through a signalling server Server.

A has a clientside heartbeat that signals its ClientState to other clients. ClientState can be one of the following:

• StateFoo
• StateBar

Sometimes, B wants to change the state of A. It maintains a representation of A locally. When it says "Change A to StateBar", it changes A.ClientState (local representation of a remote device state) locally first (for UX purposes), then signals through Server to change ClientState on A - the local state.

This usually works, but occasionally there are timing issues where the heartbeat from A will revert the manual state change that B initiated because a heartbeat was sent "sometime around" when B sent the state change signal.

I'm thinking about two approaches to solving this problem:

1. Timing - see if the server is sending something that was triggered after I locally modified state. This kind of works, but if B sends a heartbeat right after I initiate a state change from A, this will not work.
2. Command prioritization - if the last thing that changed the state was a command, wait n seconds before allowing a heartbeat (not a command) to update state again.

I don't like 2 because it involves a magic number. Network connectivity issues might push a delay past n seconds.

What kinds of patterns are out there to solve this problem? The language doesn't really matter; though I am using C#/JS/SignalR in case it matters.

• 9000: If I were to reliably replicate a state change, I'd consider the two-phase commit protocol. A heartbeat with the "old" remote state arriving between the start and completion would only confirm that the other side is alive and hasn't yet accepted the state change.

Answer 1

Keep a version number on the ClientState. Increment it anytime the state changes.

When client B sends a change to client A, B keeps record of the next version it expects from A. When it receives a heartbeat from A that has a smaller version, it can ignore the state in that heartbeat.

---

The most sure way to get around this problem is to look at the first law of distributed object design: don't distribute. Instead, make the signaling server the source of truth for all clients rather than each client being its own source of truth. Otherwise, you are facing the Byzantine Generals problem. And that is difficult to solve. Even the above suggestion doesn't cover the case where A and B decide to change A's state at the same time.