How to keep front-end models in sync with REST API/DB without any UI delay?

Question

Here's my problem. I want to make fast, subsequent requests to a REST API endpoint (/todos/:id/position) that have side effects on other resources.

As an example of 5 fast subsequent requests:

PATCH /todos/2/position {"position": 2}
PATCH /todos/2/position {"position": 3}
PATCH /todos/2/position {"position": 4}
PATCH /todos/2/position {"position": 5}
PATCH /todos/2/position {"position": 6}

As you can see, I am moving todo #2 from position 2 to position 6, one request at a time. Now of course, those calls have side effects on other todos. When todo #2 changes from position 2 to 3, the position that had position #3 goes from 3 to 2, etc. This provides some sort of race conditions, as the 5 calls are modifying data in the DB at the same time.

I have implemented transactions and ETags on this service, because at first I thought I was having concurrency/optimistic locking problem, but it seems like my problem is bigger than this. With ETAGs and transactions in place, request 1, 3 and 4 might works, but requests 2, 5 might fail, resulting my todo to only reach position 5 (and also awkwardly skipping over certain other positions). This is because request 2 does not have the right etag, because request 1 has not responded yet, etc.

How can I make sure all requests are successful and execute in the right order? Is this even possible?

Here are two solutions I have thought of and that I consider weak, since they do not resolve the root problem on the server.

1. On the front-end, have a queue of requests. Request 2 is not initiated until request 1 has resolved. Request 3 is not initiated until request 2 has resolved, etc
2. Throttle the requests. Instead of sendin 5 fast, subsequent requests, throttle them to only send the last one, i.e with a timer of 500ms.

• Euphoric: I don't see any possible solution without changing the API or the database structure.
• Maxime Dupré: What kind of API/DB change would you make?
• JimmyJames: I don't understand the use case. Why isn't this just one call to move it to the desired position? Also, it seems like you might be trying to use the position as the resource id which seems problematic.
• radarbob: Javascript Promise chaining? Programatically promises can be thened onto a previous promise, ensuring execution sequence. Illustrates the difference from a then chain all taking the same argument as returned by the initial Promise that does not guarantee sequence.
• Maxime Dupré: @JimmyJames As I wrote in my question, it could simply be one call, but then the problem still exists on the API? If other clients start using the API and they don't make 1 call, there will be the same problem.

Answer 1

You really only have a few options on how to handle something like this. The first is to change the API so that it accepts batch operations. If you had something like this:

POST /todos/position/$batch
{ 
 "operations": [
 {"id": 2, "position": 5},
 ....
 ]}

You could apply them all or rollback at one shot.

Another option is to change the client to do compensating transactions when there is a failure. This is on the client and does not guarantee strong consistency, but if you have a position change resource verb, your client could track the successes and send follow-up transactions to roll them back to the previous state. This idea is covered very well in Sagas by Hector Carcia-Molina and Kenneth Salem (https://dl.acm.org/doi/10.1145/38713.38742) and far predates the RESTful architectural style, but the idea is still there.

Finally, you could implement full two-phase commit in your REST API which is probably the hardest and least RESTful option you have.

I will probably be roasted for this part, RESTful architecture is fundamentally based on viewing your system has objects (resources, documents, etc.) but I find this doesn't always offer the lowest amount of friction when mapped to business processes. It is often more pragmatic to compromise a little bit on what makes things RESTful than it is to remain dogmatically pure about this architectural style.

• Maxime Dupré: I've thought about batching, but then it leaves the positionning logic to the client, instead of centralizing it on the server. This could lead to each client having a different implementation, which is not ideal, but maybe a sacrifice I'll have to make...I never thought making a todo application could be that hard lol!
• imel96: @maximedupre if you want to follow REST, then the client has to have some logic so it knows how to build new State to Transfer to the server
• Maxime Dupré: I think we're talking at different levels here. What I'm saying is that if I decide to batch the requests, the side-effects that are occurring on the server (namely changing the position of the other todos), would need to be replicated on all the clients. Furthermore if I batch the requests, it simply becomes a PUT /todos, which is just an update of the collection resource. What is nice about my current service is that it abstracts the side-effects from the clients.
• imel96: I think we're on the same page :) If there are side-effects on the server then for every client update (PUT/POST, etc.) the client will have to do a GET request to know what the server has done. That will be slow. I prefer the server only validates requests

Answer 2

Possible UI solution:

Block the UI interaction for that element, as long as your client app waits for the API response.

That is in my opinion much better then having a queue or threshold for subsequent api calls.

Possible API solution:

(Depends on your client app, if you can drag'n'drop your item)

Don't update each todo item as a single resource, instead add a todo list resource, which would return you this

{
 "todos": [
 {"id":1, "position": 1}
 {"id":2, "position": 2}
 {"id":3, "position": 3}
 {"id":4, "position": 4}
 ]
}

Now if you want to move Todo Item 1 to Position 4, you can simply send this to your api:

{
 "todos": [
 {"id":1, "position": 4}
 {"id":2, "position": 1}
 {"id":3, "position": 2}
 {"id":4, "position": 3}
 ]
}

• Maxime Dupré: I really don't want to block the UI, as it make the UX terrible. You don't want to wait half a second between each operation, especially a todo app where you might be pressing the arrow down a couple times to move it down two positions. As for the batching...I've thought about it, but then it leaves the positionning logic to the client, instead of centralizing it on the server. This could lead to each client having a different implementation. Maybe that's a sacrifice I need to make.
• Mischa: Maybe it's a little bit misleading, but I really don't mean batch processing.Instead, add a new resource (i.e. TodoItemsList) that contains the items and their position in that list. Then you don't have to update each item (as it does not know about position) - only the list itself. This would be a better separation of concerns.
• Mischa: Also I agree about blocking the UI, that's not really nice, though from UX perspective much better then queuing/delaying many updates. Because this brings other problems with it.

Answer 3

So, In not sure if I get what you're trying to do right, but if I understand this correctly, you have a Todo list reordering UI that is implemented by moving Todo items one position at a time, because maybe your UI used up/down arrow or something similar to reorder Todo items one item at a time, but you worry about race condition caused by your requests getting intertangled due to race condition caused by differing response times between each subsequent requests.

The simplest way to enforce this constraint is to use time-based conditional requests rather than ETag-based conditional requests. So instead of If-Match: <etag>, you'd send the current time in both the Date and If-Unmodified-Since in the request. The server should record the Date of last change against the Todo list and should conditionally perform the request only if the last updated time is older than the date in the If-Unmodified-Since header (Optionally, you can skip the Date header and just use If-Unmodified-Since for both).

That works fine if you only care about the side effect of the last reordering command, if your system somehow are made in such a way that the side effects that happens in the conditionally discarded requests are important, then you need to queue the requests either on the client or the server, depending on the consistency requirement that you're looking for.

• Maxime Dupré: I'm not sure I understand what difference it would make vs an etag. Let's take the same scenario I described in my question. The first request would be successful because the time is older. The second won't because the request has been initiated before the modification. The third has been initiated after the todo updated time, so it will be successful, etc (read full scenario in question). The last successful request might not be the last request, so the position would not be correct. Am I missing something?
• Lie Ryan: @maximedupre The timestamp is generated by the client, so as long as the client's clock is monotonously increasing, the request with the highest timestamp will always be the latest update according to the client. ETag only ensures partial ordering and the ability to pick a linear serialisation, all the other possible branches are discarded, but you can't be sure that the branch the server picked will be the one that the client sent last; timestamp based ordering ensures there is a total ordering, and the request with the highest timestamp is always the last non-discarded request.