Loading posts in real time is broken in some cases due to a breaking change in the format data is returned

Question

At some point in the last 16, or so, hours a change was made to the data returned from the Stack Exchange internal /posts/ajax-load-realtime/{list of semicolon separated postIDs} endpoint. The response changed from plain HTML text to a JSON encoded Object with at least two properties: Html and VotesCastJson. Unfortunately, when this change was made, SE's codebase wasn't searched for every place where this endpoint was used and/or those places were not actually changed.

I haven't done an exhaustive check to see everywhere which this change breaks things (e.g. there may additional things which relied upon the internal HTML structure provided). In addition, there have been reports of other issues in other areas of the site which could easily be related to this or similar endpoint changes. However, one place where this endpoint is used is StackExchange.realtime.reloadPosts([array of post IDs]). The code for that function assumes that it's still going to receive straight HTML text, rather than a JSON encoded Object with the HTML text in the Html property. This, of course, breaks everything which relies on StackExchange.realtime.reloadPosts().

You can test to see if this is working by opening up the browser console for this question, or any question, and typing StackExchange.realtime.reloadPosts([$('.question').data('questionid')]), which should reload the HTML for this question. At the time of posting this question, instead of the correct operation it will produce a Syntax Error.

Note: the new VotesCastJson property appears to be intended to reduce the need to call the /posts/${postId}/votes endpoint to get the votes for the post that's being reloaded, but the data returned in that property is invalid in all cases which I've checked.

• Sebastian Simon: Yes, I noticed this earlier today when clicking "An edit has been made to this post" didn’t do anything.
• Makyen: @Sonic, it's not fully accurate to make a blanket statement that loading posts in real time is broken. There are a few different ways to load posts in real time, and multiple places where it's done. The breaking change appears to be in only one of the endpoints and only some of the places where that endpoint is used have not been updated to the new format. So, it's not everything that's broken, just some things. For example, StackExchange.realtime.expandAnswers() was updated to use the new response format from the ajax-load-realtime endpoint, whereas .reloadPosts() was not.
• Sonic the Anonymous Hedgehog: Not a blanket statement anymore.
• Sebastian Simon: Another issue with the same cause: Post notices are not instantly displayed.
• Sebastian Simon: Now also reported on MSO: Error when loading new changes.
• Sebastian Simon: It seems this has been fixed recently, at least the "An edit has been made to this post" banners.

Answer 1

This has been fixed. This was a bug that came up during some modulification. You all are exactly right. There was a mismatch in what was being sent vs. what was expected. The original change caught it in most places, but it missed this one. This change went out on Monday.

I took a quick search through related JavaScript code and I think this was the only place that was broken. If there's more that's broken that we've missed, please let us know!

Thanks to both Makyen and VLAZ for making this easy to reproduce and fix. I really appreciate it.

Answer 2

AS OF 2022年03月30日 THE ISSUE SEEMS TO BE FIXED BY STACK EXCHANGE

THE USERSCRIPT IS NOT NEEDED ANY MORE. IT CAN BE SAFELY UNINSTALLED.

Makyen's fix does not work for me. I suspect there was an update that broke it but did not fix the behaviour. The script is correct - it monkey patches the StackExchange.realtime.reloadPosts() method, however the problem is that internally the SE code directly calls the function:

if ($('#review-content').length === 0) {
 reloadPosts([post.id]);
}

Instead of using the (overridden) method call.

(see Line 10886 in the dev distribution of full.js)

I made a fix via a different route - using $.ajaxSetup and the dataFilter property to directly get the data before it is processed and attempt to extract the HTML portion of it.

Install the userscript (Source on GitHub)

Same guidance as Makyen's:

Once SE actually fixes the issue in their code, you'll need to disable or uninstall the userscript.

Code

// ==UserScript==
// @name Quick and dirty fix for reloading posts 
// @namespace StackExchangeFix
// @description Hook directly through jQuery and Interceptdata returned from /ajax-load-realtime/ to pre-process for later calls. 
// @match https://stackoverflow.com/*
// @match *://*.stackoverflow.com/*
// @match *://*.superuser.com/*
// @match *://*.serverfault.com/*
// @match *://*.askubuntu.com/*
// @match *://*.stackapps.com/*
// @match *://*.stackexchange.com/*
// @match *://*.mathoverflow.net/*
// @exclude *://api.*
// @exclude *://blog.*
// @exclude *://chat.*
// @exclude *://data.*
// @exclude *://stackoverflow.com/advertising*
// @exclude *://stackoverflow.com/talent*
// @exclude *://stackoverflow.com/teams*
// @grant none
// @version 1.1
// @author VLAZ
// @run-at document-end
// ==/UserScript==
(function() {
 const equalCaseInsensitive = (a, b) =>
 a?.localeCompare(b, undefined, {sensitivity: "base"}) === 0;
 
 function patchReload(data, type) {
 //only intercept reloads
 if (!this.url.startsWith("/posts/ajax-load-realtime/"))
 return data;
 //these requests seem broken as they claim the result is HTML but it's actually JSON
 if (!equalCaseInsensitive(type, "html"))
 return data;
 try {
 //try to parse
 const realData = JSON.parse(data);
 //verify it's the correct shape
 if ("Html" in realData && "VotesCastJson" in realData)
 return realData.Html;
 return data;
 } catch (e) { 
 // parsing error: ignore and return the original
 return data;
 }
 }
 const prevDataFilter = $.ajaxSetup().dataFilter || (x => x);
 
 const passThroughFix = function(...args) {
 //apply fix before passing the data downstream
 const newData = patchReload.apply(this, args);
 
 const newArgs = [newData].concat(args.slice(1));
 
 //apply downstream filter if any 
 return prevDataFilter.apply(this, newArgs);
 };
 $.ajaxSetup({ dataFilter: passThroughFix });
})();

• user152859: Now I wonder who will fix your fix... ;)
• Makyen: SE uses the response from /posts/ajax-load-realtime/ both with and without assuming that it's in the new format. Thus, a global change to it, like you've done here, fixes reloadPosts() correctly, but breaks the other places where the endpoint is called. What's needed is a combination of our two approaches, so that the adjustment is only applied in the places where SE didn't change the code to use the new format response (i.e. reloadPosts()).
• VLAZ: @Makyen OK, true. I've done some testing and it seems to work but I do not have an overview of all usages of the endpoint. Assuming type is correctly "json" in the other places, it should work, since this will only work with JSON returned for type "html".
• Makyen: I mentioned it in chat, but it's something that should be known generally: For $.ajaxSetup({dataFilter:}) there can be only one dataFilter in all code on the page. If possible, a userscript should either avoid using single use resources, or account for other things (page code or other userscripts) also wanting to use that resource. In the case of dataFilter, you should acquire a reference to any existing dataFilter and have your code also pass the data through it. As with most jQuery calls, this is used here, so that needs to be set for the call to any existing dataFilter.