Skip to content

Navigation Menu

Sign in
Sign up

Lifting the JSON-serializable restriction: replacer/reviver, not a codec seam #26

AndresSaa started this conversation in Ideas
Discussion options

The problem is real

Date and BigInt are the friction people will actually hit. Today the log is
JSON, so a Date comes back as a string and a BigInt throws
ERR_ENTRY_NOT_SERIALIZABLE. That is documented, but "serialise deliberately"
only works when the payload shape is yours — it is no help when you are
forwarding a webhook body you did not author.

Why not a pluggable codec

The obvious answer is a codec seam so people can drop in superjson or CBOR. I do
not think that is a feature; I think it is a v2, because it breaks things the
package currently guarantees:

  • Framing. Recovery depends on records being newline-terminated. CBOR has no
    newline framing, so healTail — which scans backwards for the last \n
    stops working.
  • The repair procedure. docs/durability.md tells you that every line is
    independent JSON and gives you a node -e one-liner to find the bad ones. A
    binary codec makes that page wrong.
  • Validation. scanAccounting and writeSurvivors parse JSON lines to
    check that sequence numbers increase and to decide what survives compaction.

That is the on-disk format, the recovery path and the operational story, not an
extension point.

The narrower version that fits

replacer and reviver options, passed through to JSON.stringify and
JSON.parse:

const wal = createWal({
 replacer(key, value) {
 return this[key] instanceof Date ? { $date: value } : value;
 },
 reviver(key, value) {
 return value?.$date ? new Date(value.$date) : value;
 },
});

Framing is untouched — still one JSON object per line — so healing, compaction,
sequence validation and the repair procedure all keep working unchanged. No
runtime dependency either: the caller brings the transform.

Three sharp edges, verified rather than assumed

1. The replacer never sees a Date. toJSON() runs first, so the replacer
receives the ISO string. Detecting a Date requires this[key], which means the
replacer cannot be an arrow function:

valor recibido: string "1970-01-01T00:00:00.000Z" | this[k]: Date real

2. A replacer does fix BigInt. Confirmed — without one JSON.stringify
throws TypeError, with one it produces {"n":{"$bigint":"1"}}.

3. The reviver sees the envelope, not just the value. Parsing a record calls
it with the keys ["seq","d","value",""]. A careless reviver can corrupt seq.
Options: apply it to the whole line and let the existing validation reject a
mangled seq, or double-parse to isolate the value, which costs a round trip on
every read.

Open questions

  • Does a mismatched replacer/reviver pair across a restart need guarding? Today
    the on-disk format is fixed; this would make it caller-dependent, with no
    version marker in the file to catch a changed transform against old records.
  • Is edge 1 acceptable, or does the ergonomic answer need a different shape than
    raw JSON.stringify semantics?
  • Is this worth it at all, given wal.append({ at: date.toISOString() }) already
    works for payloads you control?

Not scheduled. Opening it here so the reasoning exists before anyone asks, and
so the codec version gets rejected for a stated reason rather than silently.

You must be logged in to vote

Replies: 0 comments

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Ideas
Labels
None yet
1 participant

AltStyle によって変換されたページ (->オリジナル) /