How to detect and handle changes in a realtime api

Question

Assuming I have a paginated api endpoint:

GET /api/meetings

My client is consuming this api endpoint and displays it in an infinite scroll that underneath gets page by page as the user scrolls down.

Now I want to add real time capabilities to the api. New meetings can come in and also existing meetings can change (for example, a meeting got canceled).

Ideally, I want the UI for anything I pulled so far to change if something in the meeting has changed.

Is this something that is possible to pull off in a performant way ? What changes (high level of course) do you think needs to be done to handle this ?

Thanks.

• Michael: Would be happy to know why the downvotes so I can possibly rephrase the question.
• rlanvin: I don't know about the downvote, but I can say that I don't understand how your client is consuming a paginated endpoint in "real time with SSE or websockets".
• Michael: @rlanvin that's a good point. What if you had an infinite scroll type of UI that is basically getting page after page as you scroll. Would you say you can't have updates to entries already pulled without refetching that page ?
• rlanvin: I still don't understand, does this client you are talking about actually exist (and if so, how does it work exactly because I don't get it)? Or are you just saying you think this would be a solution?
• Michael: @rlanvin I have changed the question, please let me know if it makes more sense now

Answer 1

Is this something that is possible to pull off in a performant way ?

Yes

What changes (high level of course) do you think needs to be done to handle this ?

You need a simple pub/sub mechanism.

• Your client subscribes to notifications from the server about creation and modification of meetings
• Every time a meeting is created or modified, the server publishes a notification to all the subscribers
• When a client receives a notification about a meeting, it can decide whether to append it to the list, or replace an existing meeting if it was already in the list (already fetched using the GET endpoint) or simply ignore it if the meeting is not part of the list being displayed.

With a frontend (javascript) client, which I think is your case, you can use a websocket connection (or SSE) to a broker. The client connects to a websocket/SSE service (the broker) and then the server publishes notification to all the clients connected when a meeting is created/updated. There are many open source servers available for this purpose, as well as commercial hosted solutions (SaaS).

With a backend client, the subscription can be made with a HTTP request, e.g. POST /api/meetings/subscribe in which the client provides a callback URL. The server publishes events by making a GET request to all the callback URLs that have been provided, also known as "webhook".

• Michael: Interesting. I assume you're talking about something like pushpin.org. What is the benefit of using such broker as opposed to implementing the push on the server side directly ?
• rlanvin: I didn't know pushpin.org but yes there are plenty of options to choose from. As for using a broker it's up to you and your technology stack. I personally prefer service oriented architecture over big monolithic application, but that's an entirely different topic. The point is: use a pub/sub mechanism to push notifications in real time to the clients and it will be efficient.

Answer 2

Your client needs to pass one or more parameters to tell the server which meetings it has and which versions of those meetings.

A very simple way to do this is to include a "lastUpdated" field in your meeting object. Any time you change or create a meeting, update that field with the current date/time.

When your client calls the server, it can pass in a parameter that says "I already have meetings with a lastUpdated field < xxxx". The server would then reply with any updates as well as any new records.

• Michael: How does this work when the api is paginated ? This will have to be in a separate api endpoint I assume ?
• Dan Pichelman: What do you mean by paginated? Are you passing a parameter to the server "give me page 1"?
• Michael: Yes. API is paginated and the UI shows an infinite scroll that is basically fetching page by page when the user scrolls down
• Dan Pichelman: So instead of asking for "page 6", ask for "all updated since 2018年1月8日 14:01Z"