Full Replication #20

Architecture

We adopt a leader-less architecture, where every server can receive writes from clients, and these writes will be replicated to configured replicas. In other words, every server can be a leader, and/or a replica. Note that writes from a leader will not be replicated to other replicas (the max level of replication is 1).

Leader Design

Data received from clients will be appended into one or more global, file-backed queues. Each item in these queues will be labeled by a "check-point". There will be exactly one connection from the leader to a replica for each of these queues.

The first time a leader connect to a replica, it should receive a message from the replica indicating the last "check-point" that it received (and persisted, of course). The leader will then start forwarding data starting at that "check-point".

Periodically a replica will flush its data from mmap file to disk. After the flush, the replica will send the latest check-point that got persisted on disk back to the leader, who can then free all the items in the queue up to that check-point, assuming ALL other replicas, if any, have also done so.

Replica Design

When a new connection comes in, from a leader, the very first line of data will indicate that the connection came from a leader. It will then send the last persisted check-point back to the leader.

After a periodic flush, the replica will also persist the latest check-point of each connection that came from a leader. And, it will send the check-point back to the leader, via the same connection, so that the leader can discard any data that's already persisted on this replica.

Add a Replica

To add a brand new replica, find a server, preferably a replica, to copy the data from. First shutdown the server; then copy all the data files over; then start/restart both. You will need to update the config of the leader with the additional information about the new replica before all of these.