Crash when /system/sitemap.json returns HTML (uncaught SHASHT parse error) #4

rgb commented 2026年02月15日 10:29:13 +01:00

When HyperDoc’s fedwiki catalog loader fetches remote .../system/sitemap.json, it assumes the response is JSON and calls shasht:read-json unconditionally. If a remote site returns an HTML error page (first non-whitespace char is <), SHASHT signals SHASHT-INVALID-CHAR and the condition is not caught by the handler-case in the background thread. With --disable-debugger, this aborts the process and brings down the server.

This is easy to hit with transient upstream failures, redirects, captive portals, misconfigured sites, or "friendly" error pages.

Environment

• SBCL 2.4.10 (reproduced on macOS + Nix dev shell)
• HyperDoc running via ./start.sh
• CLOG + Hunchentoot + Clack/Hunchentoot handler (as currently shipped in this repo)

Steps to reproduce

1. Start server: ./start.sh

2. Ensure fedwiki integration loads one or more sites where http://<site>/system/sitemap.json returns HTML (e.g., an error page, proxy page, redirect page).

   • In my logs, examples were:

     • http://ims.obeya.xyz/system/sitemap.json
     • http://marc.obeya.xyz/system/sitemap.json

3. Observe server abort shortly after "New connection id ...".

Actual behavior
Server aborts with an unhandled condition in a background thread, then fails to shut down cleanly.

Representative log excerpt

Unhandled SHASHT:SHASHT-INVALID-CHAR ...:
Unexpected character #< found during parsing.
...
(HYPERBOOK/FEDWIKI::FETCH-JSON "http://ims.obeya.xyz/system/sitemap.json")
(HYPERBOOK/FEDWIKI::FETCH-SITEMAP ...)
...
unhandled condition in --disable-debugger mode, quitting

Expected behavior

• Remote fetch failures (including non-JSON responses) should not crash HyperDoc.
• The failing wiki entry should be marked as failed/unreachable with a stored status condition, and the server should continue running.

Probable cause

• fetch-json calls shasht:read-json on the HTTP stream without validating content-type or first character.
• make-fedwiki wraps the refresh in a handler-case that catches only stream/usocket/drakma-ish conditions but does not catch parser errors (shasht:*) or generic error.

Proposed fix (minimal)

1. Extend make-fedwiki’s handler-case with a final (error (c) (setf (status-of wiki) c)) clause so unexpected conditions never abort the process.
2. Improve fetch-json diagnostics by peeking the first non-whitespace character and raising a clear error when the response starts with < (HTML), e.g. "Expected JSON from ... but got HTML".

Notes
This is not about "bad data" per se; it’s about robustness: remote federation endpoints can and do return HTML in error modes, and we should treat that as a normal failure case, not a fatal crash.

When HyperDoc’s fedwiki catalog loader fetches remote `.../system/sitemap.json`, it assumes the response is JSON and calls `shasht:read-json` unconditionally. If a remote site returns an HTML error page (first non-whitespace char is `<`), SHASHT signals `SHASHT-INVALID-CHAR` and the condition is not caught by the `handler-case` in the background thread. With `--disable-debugger`, this aborts the process and brings down the server. This is easy to hit with transient upstream failures, redirects, captive portals, misconfigured sites, or "friendly" error pages. Environment * SBCL 2.4.10 (reproduced on macOS + Nix dev shell) * HyperDoc running via `./start.sh` * CLOG + Hunchentoot + Clack/Hunchentoot handler (as currently shipped in this repo) Steps to reproduce 1. Start server: `./start.sh` 2. Ensure fedwiki integration loads one or more sites where `http://<site>/system/sitemap.json` returns HTML (e.g., an error page, proxy page, redirect page). * In my logs, examples were: * `http://ims.obeya.xyz/system/sitemap.json` * `http://marc.obeya.xyz/system/sitemap.json` 3. Observe server abort shortly after "New connection id ...". Actual behavior Server aborts with an unhandled condition in a background thread, then fails to shut down cleanly. Representative log excerpt Unhandled SHASHT:SHASHT-INVALID-CHAR ...: Unexpected character #< found during parsing. ... (HYPERBOOK/FEDWIKI::FETCH-JSON "[http://ims.obeya.xyz/system/sitemap.json](http://ims.obeya.xyz/system/sitemap.json)") (HYPERBOOK/FEDWIKI::FETCH-SITEMAP ...) ... unhandled condition in --disable-debugger mode, quitting Expected behavior * Remote fetch failures (including non-JSON responses) should not crash HyperDoc. * The failing wiki entry should be marked as failed/unreachable with a stored status condition, and the server should continue running. Probable cause * `fetch-json` calls `shasht:read-json` on the HTTP stream without validating content-type or first character. * `make-fedwiki` wraps the refresh in a `handler-case` that catches only stream/usocket/drakma-ish conditions but does not catch parser errors (`shasht:*`) or generic `error`. Proposed fix (minimal) 1. Extend `make-fedwiki`’s `handler-case` with a final `(error (c) (setf (status-of wiki) c))` clause so unexpected conditions never abort the process. 2. Improve `fetch-json` diagnostics by peeking the first non-whitespace character and raising a clear error when the response starts with `<` (HTML), e.g. "Expected JSON from ... but got HTML". Notes This is not about "bad data" per se; it’s about robustness: remote federation endpoints can and do return HTML in error modes, and we should treat that as a normal failure case, not a fatal crash.

rgb commented 2026年02月15日 10:39:38 +01:00

Smoke Test: Non-JSON /system/sitemap.json does not crash server

Purpose

Verify that:

1. MAKE-FEDWIKI does not crash when a remote site returns HTML instead of JSON.
2. status-of contains a condition.
3. The SBCL process exits normally (exit code 0 in this test).

---

Environment

• SBCL 2.4.10

• Nix dev shell (nix develop)

• Systems loaded:

  • :hyperbook/server
  • :hyperbook/fedwiki

---

Test Command

Run:

nix develop -c sbcl --no-userinit --non-interactive \
 --eval '(require :asdf)' \
 --eval '(asdf:load-system :hyperbook/server)' \
 --eval '(asdf:load-system :hyperbook/fedwiki)' \
 --eval '(let* ((pkg (find-package "HYPERBOOK/FEDWIKI"))
 (make-fedwiki (symbol-function (find-symbol "MAKE-FEDWIKI" pkg)))
 (status-of-sym (find-symbol "STATUS-OF" pkg))
 (wiki (funcall make-fedwiki "example.com"))
 (st (funcall status-of-sym wiki)))
 (when (typep st (quote bt:thread))
 (bt:join-thread st)
 (setf st (funcall status-of-sym wiki)))
 (format t "STATUS-TYPE=~A~%" (type-of st))
 (format t "IS-CONDITION=~A~%" (typep st (quote condition)))
 (format t "IS-T=~A~%" (eq st t))
 (when (typep st (quote condition))
 (format t "COND=~A~%" st))
 (uiop:quit (if (typep st (quote condition)) 0 1)))'

---

Expected Output (Observed)

STATUS-TYPE=SIMPLE-ERROR
IS-CONDITION=T
IS-T=NIL
COND=Expected JSON from http://example.com/system/sitemap.json but got HTML (starts with '<').

Exit code: 0

---

What This Confirms

✔ HTML response is detected
✔ A condition object is stored in status-of
✔ No unhandled SHASHT-INVALID-CHAR
✔ No background thread abort
✔ No server crash

---

Regression Target

Previously this scenario resulted in:

Unhandled SHASHT:SHASHT-INVALID-CHAR
Unexpected character #\< found during parsing.
...
unhandled condition in --disable-debugger mode, quitting

which terminated the process.

---

Success Criterion

The system must:

• Contain the error as a condition
• Remain running
• Never propagate parser errors past make-fedwiki

---

## Smoke Test: Non-JSON `/system/sitemap.json` does not crash server ### Purpose Verify that: 1. `MAKE-FEDWIKI` does **not crash** when a remote site returns HTML instead of JSON. 2. `status-of` contains a `condition`. 3. The SBCL process exits normally (exit code `0` in this test). --- ## Environment * SBCL 2.4.10 * Nix dev shell (`nix develop`) * Systems loaded: * `:hyperbook/server` * `:hyperbook/fedwiki` --- ## Test Command Run: ```bash nix develop -c sbcl --no-userinit --non-interactive \ --eval '(require :asdf)' \ --eval '(asdf:load-system :hyperbook/server)' \ --eval '(asdf:load-system :hyperbook/fedwiki)' \ --eval '(let* ((pkg (find-package "HYPERBOOK/FEDWIKI")) (make-fedwiki (symbol-function (find-symbol "MAKE-FEDWIKI" pkg))) (status-of-sym (find-symbol "STATUS-OF" pkg)) (wiki (funcall make-fedwiki "example.com")) (st (funcall status-of-sym wiki))) (when (typep st (quote bt:thread)) (bt:join-thread st) (setf st (funcall status-of-sym wiki))) (format t "STATUS-TYPE=~A~%" (type-of st)) (format t "IS-CONDITION=~A~%" (typep st (quote condition))) (format t "IS-T=~A~%" (eq st t)) (when (typep st (quote condition)) (format t "COND=~A~%" st)) (uiop:quit (if (typep st (quote condition)) 0 1)))' ``` --- ## Expected Output (Observed) ``` STATUS-TYPE=SIMPLE-ERROR IS-CONDITION=T IS-T=NIL COND=Expected JSON from http://example.com/system/sitemap.json but got HTML (starts with '<'). ``` Exit code: `0` --- ## What This Confirms ✔ HTML response is detected ✔ A condition object is stored in `status-of` ✔ No unhandled `SHASHT-INVALID-CHAR` ✔ No background thread abort ✔ No server crash --- ## Regression Target Previously this scenario resulted in: ``` Unhandled SHASHT:SHASHT-INVALID-CHAR Unexpected character #\< found during parsing. ... unhandled condition in --disable-debugger mode, quitting ``` which terminated the process. --- ## Success Criterion The system must: * Contain the error as a condition * Remain running * Never propagate parser errors past `make-fedwiki` ---

rgb commented 2026年02月19日 12:05:38 +01:00

• Branch/commit tested: mbp-bs @ ba65397
• Inspection result: hyperbook-fedwiki/fedwiki.lisp:206 now guards JSON parsing by sniffing < before shasht:read-json, and make-fedwiki wraps sitemap/plugin fetch in handler-case and stores condition in status instead of crashing (hyperbook-fedwiki/fedwiki.lisp:140).
• Forced repro setup: local bad upstream at 127.0.0.1:19090 returning 200 text/html and <!doctype html>... for /system/sitemap.json.
• Forced repro result: direct call to hyperbook/fedwiki::fetch-json against that endpoint produced a handled error, not an uncaught SHASHT crash:
  • caught-error=Expected JSON from http://127.0.0.1:19090/system/sitemap.json but got HTML (starts with '<').
  • request log confirmed /system/sitemap.json was hit.

Decision: rgb/hyperdoc#4 is fixed (for the HTML-instead-of-JSON crash path).
The current behavior is fail-fast with a readable error, and the fedwiki loader path is structured to degrade via captured status rather than process crash.

- Branch/commit tested: mbp-bs @ ba65397 - Inspection result: hyperbook-fedwiki/fedwiki.lisp:206 now guards JSON parsing by sniffing < before shasht:read-json, and make-fedwiki wraps sitemap/plugin fetch in handler-case and stores condition in status instead of crashing (hyperbook-fedwiki/fedwiki.lisp:140). - Forced repro setup: local bad upstream at 127.0.0.1:19090 returning 200 text/html and <!doctype html>... for /system/sitemap.json. - Forced repro result: direct call to hyperbook/fedwiki::fetch-json against that endpoint produced a handled error, not an uncaught SHASHT crash: - caught-error=Expected JSON from http://127.0.0.1:19090/system/sitemap.json but got HTML (starts with '<'). - request log confirmed /system/sitemap.json was hit. Decision: rgb/hyperdoc#4 is fixed (for the HTML-instead-of-JSON crash path). The current behavior is fail-fast with a readable error, and the fedwiki loader path is structured to degrade via captured status rather than process crash.