-
Notifications
You must be signed in to change notification settings - Fork 35
feat: add "comic" source type for daily comic strips (XKCD, gocomics.com, arcamax.com) - #133
feat: add "comic" source type for daily comic strips (XKCD, gocomics.com, arcamax.com) #133Smengerl wants to merge 12 commits into
Conversation
Smengerl
commented
Aug 3, 2026
...status feature/section-provider was missing from the PR-tracking table entirely. feature/comic-provider's row still said "not yet opened" - PR j6k4m8#133 has actually been open since 2026年08月03日.
Smengerl
commented
Aug 9, 2026
Update: reworked this to be much more generic. Instead of one hardcoded comic_type per specific comic (cah, garfield), gocomics/arcamax are now generic site-backed sources that take any comic's own slug via a new comic_name field (e.g. {"comic_type": "gocomics", "comic_name": "beetlebailey"}). Both sites serve their entire catalog (hundreds of strips) through one identical URL/markup template, so this unlocks any comic on either site with no code change - verified live against several strips per site. Headline is now derived straight from the fetched page (gocomics.com's JSON-LD, arcamax.com's og:title) instead of a hardcoded per-comic label. xkcd is unchanged.
Added reliability fix for gocomics.com: if a comic's strip for the requested day isn't published yet, it now silently falls back to the most recent day that is available (up to a few days back) rather than dropping the section. The reader may briefly see a repeat of an older strip instead of that day's - only a log line marks this, nothing shows up differently in the newspaper itself.
20471af to
e58106c
Compare
Pulls the Pillow re-encode step (bound dimensions, normalize color mode, composite transparency, always emit JPEG) out into storyprovider/imageutil.py, generic over any source of fetched image bytes - not RSS-specific. This is designed to land the same way in the comic-provider PR (j6k4m8#133), which needs the identical re-encode step for comic strip images and previously had its own separate, bespoke copy of this logic. Sharing one module lets both PRs merge in either order: each carries its own copy of imageutil.py (unavoidable, since neither branch can depend on the other not being merged first), which is at worst a trivial identical-content conflict for whichever merges second - same category of expected mechanical overlap already called out for this PR's config schema changes. Also adds direct unit tests for the re-encode step itself (test_imageutil.py), decoupled from HTTP mocking - the RSS-specific wiring tests (does _inline_remote_images fetch/skip/tolerate-failure correctly) stay in test_rss.py. 🤖 Generated with Claude Code
Smengerl
commented
Aug 11, 2026
Update: extracted the Pillow re-encode step (bound dimensions, normalize color mode, composite transparency, always JPEG) out of this file into a new storyprovider/imageutil.py, shared with a sibling fix (RSS-sourced article images need the exact same defensive treatment, for the exact same WeasyPrint-silently-drops-the-story reason - not yet a PR here, but landing the same way). No behavior change here, just factoring out logic that's no longer comic-specific. Both PRs carry their own copy of imageutil.py so either can merge first without depending on the other.
Smengerl
commented
Aug 11, 2026
The sibling fix mentioned above is now #142.
Same pattern the maintainer suggested for comic.py's comic_type in j6k4m8#133. Mirrors the "kept in sync by hand" comment convention already used there.
...ide convention Follows the same format j6k4m8#140 established for the other built-in providers (and that j6k4m8#133 already used for comic.py.md). README link deliberately left out for now - the README section is one of the files already conflicting with master, better resolved together.
- comic.py's class docstring and a test_comic.py comment had reintroduced the "verified live" debug-log-style anecdotal-claim pattern that a commit already on this branch (d4fcde5) had deliberately removed from this same file per maintainer feedback on j6k4m8#141 - almost certainly pasted back in from an earlier draft of this docstring written before that fix existed. Restated the same facts without the anecdotal framing. - docs/reference/storyprovider/comic.py.md (added by j6k4m8#133, not touched by the previous commit) still said the strip image "is downloaded, decoded, and re-encoded as JPEG... not linked by remote URL" - exactly backwards after this branch's change. Updated to describe the current behavior (left as a remote link; fetch/ normalize/inline happens centrally in Goosepaper). - Added an end-to-end test running the actual DailyComicStoryProvider.get_stories() (not a hand-built stand-in for its <style>+<div> shape) through a real Goosepaper.to_html() - matching the treatment the Readwise integration test already had, which comic.py's own image-inlining path was missing. 145/145 tests pass.
Downloads today's XKCD, Calvin and Hobbes, or Garfield strip and embeds it as a single image Story. The fetch mechanism - page URL, per-comic request headers, and the XPath used to locate the strip's <img> tag - is ported from evidlo/remarkable_news's systemd service definitions (services/ xkcd.service, cah.service, garfield.service), which use the same approach to push comics onto a reMarkable's suspend screen. The downloaded strip is inlined as a base64 data: URI rather than linked by remote URL: gocomics.com requires the same browser-like headers for the image request as for the page request, and WeasyPrint (which fetches <img src> URLs itself while rendering the PDF) has no way to attach them. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Embedding the raw fetched bytes directly as a base64 data: URI seemed fine in isolation, but broke silently on the real Garfield source: arcamax.com serves a CMYK JPEG with a large embedded Photoshop/ICC metadata block, and passing that straight to WeasyPrint made it drop the *entire* story with no exception and no log line - just an empty gap where "Comics" should have been, confirmed only by diffing rendered PDF text against the source HTML. Decode with Pillow and re-encode as a clean PNG before embedding, converting to RGB/L first when the source isn't already one of those modes (handles CMYK and any other decode-only mode a comic site might serve). This mirrors remarkable_news's own Go tool, which never embeds fetched bytes directly either - it decodes then re-encodes via imaging.Decode/imaging.Save. Pillow was already a transitive dependency via weasyprint; now declared directly since comic.py imports it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The Pillow re-encode from the previous commit fixed the CMYK/metadata case but not the general one: gocomics.com's CDN can serve a strip at 2800px+ wide with no smaller variant requested, and even at a source's default resolution, a lossless PNG re-encode of a dithered/gradient-heavy color strip is several times larger than the same content as JPEG. Either way the resulting base64 payload (multi-hundred-KB, sometimes >1MB for a single story) combined with the hundreds of other images already in a full newspaper was enough to make WeasyPrint silently drop the story entirely - confirmed by bisecting a real "Julians Zeitung" generation down to the image size specifically, after ruling out layout, section ordering, and provider wrapping as causes. Cap the long edge to _MAX_IMAGE_DIMENSION (1200px) and switch the re-encode target from PNG to JPEG (quality=90) - matching remarkable_news's own Go tool, which resizes to the target screen size and always saves via imaging.JPEGQuality, never PNG. Verified against all three live sources: body_html size dropped from >1MB to ~300KB (Calvin and Hobbes) and ~125KB (Garfield). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Previously: XKCD's headline was the strip's own per-day title (from its <img> alt text), while Garfield/Calvin and Hobbes fell back to "<name> - <date>" - and every comic additionally set byline=<name>. Two problems in practice: - XKCD showed headline "Main Span" with byline "XKCD" right underneath it - looks like a subheading, but for a single-panel strip there's no "subheading" to show, just the same source name the reader already knows from the section it's in. - Garfield/CaH's "<name> - <date>" headline duplicated the name that was *also* set as the byline right below it - e.g. headline "Garfield - August 02, 2026" directly above byline "Garfield" is the same identifier twice in a two-line block, worse when two comics share one "Comics" section and both do this right next to each other. Now every comic gets a single, fixed headline - "XKCD", "Garfield", or "Calvin and Hobbes" - and no byline. Unlike an RSS article, where the byline distinguishes otherwise-anonymous entries pulled from different feeds into one section, a comic's headline already *is* that identifier - a byline under it, or a per-day headline that just restates it, adds nothing. XKCD's own per-day title/mouseover joke aren't lost: the title still ends up in the embedded <img>'s alt attribute, and the joke still renders as a caption underneath, exactly as before - only the *headline* source changed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Pillow's Image.convert("RGB") does not composite transparent pixels against
anything - it just drops the alpha channel and keeps whatever RGB value (or,
for a GIF's transparency-color-key, whatever palette color) was stored
underneath. Verified directly against Pillow: a semi-transparent black RGBA
pixel converts to solid black, not white; a color-keyed "transparent" GIF
pixel converts to its own arbitrary palette color. Applied unconditionally,
this could leave visible phantom colors/edges wherever a source image used
transparency - not observed against any of the three real sources today (all
consistently serve opaque images), but a real gap for a hypothetical source
that does.
Composite onto white before dropping alpha whenever the source image has any
transparency (RGBA/LA/PA mode, or a GIF-style transparency color-key) - white
because a comic strip always sits on a plain newspaper page, and every
bundled goosepaper style renders that page white.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
pyproject.toml has declared pillow directly (see the earlier "comic" source type commit) but uv.lock's own per-package listing for goosepaper never picked it up as a direct dependency - only the transitive entry already present via weasyprint existed. Regenerated via `uv sync`. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replaced the two site-specific, hardcoded comic_type entries ("cah" for
Calvin and Hobbes, "garfield") with two generic site-backed ones:
comic_type "gocomics"/"arcamax" plus a required comic_name (the comic's
own slug on that site, e.g. "garfield" or "beetlebailey"). Both sites
serve every comic in their catalog through one identical URL/markup
template - only the slug differs - so this was verified live against
several strips on each site rather than assumed.
The one wrinkle this raises - a fixed per-comic headline no longer works
once one _ComicSource entry covers hundreds of comics - is solved by
deriving the label straight from the fetched page itself: gocomics.com's
schema.org JSON-LD ComicSeries name, arcamax.com's og:title meta tag.
No hardcoded label table to maintain as comics get added.
Also folds in a since-diagnosed reliability fix for the date-scoped
gocomics.com URL: generation running earlier in the day than the site's
(undocumented) daily rollover was failing outright every time. Rather
than guess the rollover's timezone (an earlier version of this commit
assumed US-Eastern - wrong, and unverifiable either way), it now retries
up to a few days back on a miss and logs when that happens, instead of
dropping the section. This now benefits every gocomics comic, not just
Calvin and Hobbes.
xkcd is unchanged - it only ever serves one comic.
🤖 Generated with Claude Code
Follows the same per-provider reference doc convention master now uses for every other built-in source (see docs: add story provider guides, upstream a556ff9) instead of the inline "## comic source options" README section this branch carried before rebasing onto that change.
Addresses j6k4m8's review comment on this PR: type comic_type as Literal["xkcd", "cah", "garfield"] so editors can hint/autocomplete it. Same idea, updated to the current three values (xkcd/gocomics/arcamax) after the generic-by-slug redesign. Kept as a manually-declared alias next to _COMIC_SOURCES rather than derived from its keys - Literal can't be built from a dict at type-check time (his other suggestion, which he'd flagged as untested). The runtime membership check in __init__ stays: Literal only helps callers written directly in Python, not config-driven ones (see util.py) that pass a plain str straight from JSON.
Replaces comic.py's own inline Pillow re-encode block (bound dimensions, normalize color mode, composite transparency, always JPEG) with a call to the new shared storyprovider/imageutil.py - identical logic, now factored out since RSS-sourced article images need the exact same defensive treatment for the exact same reason (fix/rss-image-embedding, not yet upstreamed). Both PRs carry their own copy of imageutil.py so either can merge first without depending on the other - at worst a trivial identical- content conflict for whichever merges second, same category of expected mechanical overlap already called out for this PR's config.py/util.py registry changes. No behavior change: same tests (comic image processing already had dedicated CMYK/transparency/oversized-image regression tests, now also exercised directly against imageutil in test_imageutil.py), verified live against arcamax.com/gocomics.com. 🤖 Generated with Claude Code
Same issue the maintainer flagged in j6k4m8#141 for rss.py - an unverifiable anecdotal claim in a docstring rather than documentation.
d4fcde5 to
04f94cd
Compare
Same pattern the maintainer suggested for comic.py's comic_type in j6k4m8#133. Mirrors the "kept in sync by hand" comment convention already used there.
...ide convention Follows the same format j6k4m8#140 established for the other built-in providers (and that j6k4m8#133 already used for comic.py.md). README link deliberately left out for now - the README section is one of the files already conflicting with master, better resolved together.
Pulls the Pillow re-encode step (bound dimensions, normalize color mode, composite transparency, always emit JPEG) out into storyprovider/imageutil.py, generic over any source of fetched image bytes - not RSS-specific. This is designed to land the same way in the comic-provider PR (j6k4m8#133), which needs the identical re-encode step for comic strip images and previously had its own separate, bespoke copy of this logic. Sharing one module lets both PRs merge in either order: each carries its own copy of imageutil.py (unavoidable, since neither branch can depend on the other not being merged first), which is at worst a trivial identical-content conflict for whichever merges second - same category of expected mechanical overlap already called out for this PR's config schema changes. Also adds direct unit tests for the re-encode step itself (test_imageutil.py), decoupled from HTTP mocking - the RSS-specific wiring tests (does _inline_remote_images fetch/skip/tolerate-failure correctly) stay in test_rss.py. 🤖 Generated with Claude Code
Follows a suggestion from @jpco on j6k4m8#142: image size/format normalization belongs in rendering code, which knows the actual page_profile/layout being rendered, rather than being opt-in per story provider with a guessed size constant. Moves the image re-encode step out of RSSFeedStoryProvider and DailyComicStoryProvider (added in j6k4m8#142/j6k4m8#133) and into Goosepaper itself: - New _image_max_dimension() derives the embedded-image pixel cap from the actual page geometry (page_profile's content width / effective_columns * a target DPI), instead of the flat 1200 both providers guessed at independently. - New _inline_story_images() walks a story's body_html and fetches/ decodes/normalizes any <img> it finds - remote http(s):// or an already-inlined data: URI. _inline_all_story_images() runs it over every story, isolating one story's failure from the rest. - _render_html_document() (used by to_html()/to_pdf()) and to_epub() both call it now. to_epub() previously got no image protection at all from either j6k4m8#142 or j6k4m8#133 - it now does too, sized with a flat fallback constant since there's no page_profile/layout concept for a reflowable epub. - RSSFeedStoryProvider no longer fetches/re-encodes images - leaves <img src> exactly as the source article served it. - DailyComicStoryProvider no longer fetches the strip image at all - only resolves which URL is the actual strip and leaves it as a remote link, same as RSS. image_url is escaped before embedding (a source page's own src attribute value, already entity-decoded by lxml, is otherwise attacker-controlled markup). Net effect: every story provider gets the same image protection uniformly (previously only RSS and comics were covered, each with its own copy of the size-capping logic), and images are sized for the page they're actually rendered on instead of a guess. Verified against a real daily edition (26 RSS feeds + XKCD + puzzles + weather/wikipedia, page_profile: paper_pro, 2-column): identical story/page/image-inlining counts vs. the current per-provider handling, ~40% smaller total embedded image payload. Depends on j6k4m8#142 and j6k4m8#133 - built on top of imageutil.py and the per-provider embedding they added. Known limitations, not hidden: - A story whose image fails to fetch/decode fails soft (keeps the original, unprocessed link) rather than the story being dropped - matches RSS's existing behavior; comic strips previously had a stronger guarantee at the provider level that this removes. - Every image in a document gets the same size cap regardless of placement (ear/sidebar vs. main column) - would need styles.py changes to do properly. - to_epub()'s fallback size is a flat constant, not per-device - real support for j6k4m8#143's use case would need a target-device parameter threaded through. 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com>
- comic.py's class docstring and a test_comic.py comment had reintroduced the "verified live" debug-log-style anecdotal-claim pattern that a commit already on this branch (d4fcde5) had deliberately removed from this same file per maintainer feedback on j6k4m8#141 - almost certainly pasted back in from an earlier draft of this docstring written before that fix existed. Restated the same facts without the anecdotal framing. - docs/reference/storyprovider/comic.py.md (added by j6k4m8#133, not touched by the previous commit) still said the strip image "is downloaded, decoded, and re-encoded as JPEG... not linked by remote URL" - exactly backwards after this branch's change. Updated to describe the current behavior (left as a remote link; fetch/ normalize/inline happens centrally in Goosepaper). - Added an end-to-end test running the actual DailyComicStoryProvider.get_stories() (not a hand-built stand-in for its <style>+<div> shape) through a real Goosepaper.to_html() - matching the treatment the Readwise integration test already had, which comic.py's own image-inlining path was missing. 145/145 tests pass.
Every DailyComicStoryProvider story already got a fixed, source-derived headline (the comic's name, e.g. "Garfield") - kept for the table of contents, anchor-id slugging, and dedup, but rendered as a real, always- visible <h1 class="story-headline"> above the strip too. That's a plain duplicate: the strip already shows its own name inside the image, so repeating it as running text above adds nothing - unlike a headline/byline on an RSS article, which names something the fetched content doesn't already show on its own. Adds Story.headline_visible (default True, no behavior change for every other story type) - to_html() only emits the <h1>/<h2>/... tag when it's true; the headline text itself is untouched either way, so everything else that reads it directly still works exactly as before (table-of- contents entries, anchor ids, deduplicate=True's headline-based matching). DailyComicStoryProvider now always constructs its Story with headline_visible=False - a fixed property of every comic, not a config- exposed option, per the class docstring's now-updated rationale. Verified live end to end against a real XKCD fetch: table of contents still lists "XKCD" and links to the strip; the strip's own page shows only the image and its mouseover-joke caption, no headline above it. Full suite: 147 passed (143 previously + 4 new: goosepaper/test_story.py, covering the new default/override/anchor-id-still-works behavior directly, plus 2 existing comic tests (xkcd, gocomics) now also assert headline_visible is False on the returned Story). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
Motivation
A comics section is a staple of just about every physical newspaper - and every developer loves
XKCD. This adds a native
"comic"source type so a goosepaper edition can include one, withoutneeding a screen-scraping RSS-hack workaround.
What it does
Downloads today's strip and embeds it as a single image story:
{ "type": "comic", "comic_type": "xkcd" }{ "type": "comic", "comic_type": "gocomics", "comic_name": "calvinandhobbes" }comic_typexkcd,gocomics,arcamax. No default on purpose - a config that forgets it fails loudly instead of silently always fetching XKCD.comic_namegocomics/arcamax, rejected forxkcd"garfield"or"calvinandhobbes"for gocomics.com,"beetlebailey"for arcamax.com.gocomics.comandarcamax.comeach serve every comic in their catalog (hundreds of strips)through one identical URL/markup template - only the slug differs. So rather than one hardcoded
comic_typeper specific comic, those two are generic site-backed sources: any comic hosted oneither site works just by passing its slug as
comic_name, verified live against severaldifferent strips on each site.
xkcdonly ever serves one comic and takes nocomic_name.The headline is a fixed, source-derived name, never the strip's own per-day title, and no byline
is set - a byline or a per-day dynamic headline would just repeat the same source name the
headline already shows, or produce a needlessly long heading, unlike a byline on an RSS article
(which distinguishes otherwise-anonymous entries pulled from different feeds). For
xkcdit'sthe fixed string
"XKCD"; its real per-day title is still available via the embedded image'saltattribute, and its mouseover joke still renders as a caption underneath. Forgocomics/arcamax- where one source entry covers hundreds of possible comics, so there's noper-comic label to hardcode - the headline is instead read straight off the fetched page itself:
gocomics.com's structured
ComicSeriesJSON-LD, arcamax.com'sog:titlemeta tag.gocomics.com's page URL is date-scoped (
.../<comic_name>/YYYY/MM/DD). Its dailypublish-rollover time/timezone isn't documented, so a generation run earlier in the day than
that rollover would otherwise fail outright every time; rather than guess at a timezone, a miss
now retries a few days back automatically before giving up; a log line notes it, nothing shows
up differently in the newspaper itself.
The strip image is downloaded, decoded, and re-encoded (not linked by remote URL and not passed
through unmodified) via Pillow:
exception, no log line - for certain source images (a CDN serving print resolution up to
2800px wide with no smaller variant, a lossless PNG re-encode of gradient-heavy color art being
several times larger than the same content as JPEG, and CMYK-mode JPEGs with large embedded
Photoshop/ICC metadata all independently triggered it). Bounding pixel dimensions, normalizing
color mode, and using JPEG's lossy compression instead of PNG fixes all three.
Pillow's default behavior (keep whatever RGB value was stored under a transparent pixel,
unchanged) show phantom colors - not observed against any real source tested so far, but a
real gap for a hypothetical source that does use transparency.
data:URI (rather than a remote<img src>) also sidesteps gocomics.comrequiring the same browser-like headers for the image request as for the page request -
WeasyPrint fetches
<img src>URLs itself while rendering the PDF, with no way to attach them.Demo
Rendered (with the original
xkcd/cah/garfieldversion of this PR, before the slug-genericrework above - the embedding/rendering pipeline itself is unchanged) across a few different
bundled styles/layouts/page profiles, to show it holds up across genuinely different goosepaper
configurations - not just one specific theme:
remarkable2- a compact single-column e-reader layout.paper_pro- side-by-side comics reading like adjacent newspaperboxes.
a4- a dense, print-style multi-column page.Happy to paste screenshots in here directly if useful - just say the word.
Testing
uv run pytest), including dedicated regression tests for: theimage-embedding bug and its three contributing causes, the transparency-compositing fix, an
unknown
comic_typebeing rejected,comic_namebeing required/rejected percomic_type,headline derivation from each site's own page markup, and the gocomics.com retry-on-miss
behavior.
several different comic slugs on gocomics.com/arcamax.com, as of this writing.
flake8clean (both the syntax-error pass and the full statistics pass CI runs).🤖 Generated with Claude Code
Merge overlap note
Verified by locally merging every pairwise combination of my currently open PRs against
master. This one produces a mechanical (non-semantic) merge conflict with #121, #123, #124, #132 - all add a new source type/fields to the same shared registries inconfig.py/util.py. No logic overlap in any case checked.