-
Notifications
You must be signed in to change notification settings - Fork 35
Add native content_filters/skip_title_patterns to "rss" sources - #121
Add native content_filters/skip_title_patterns to "rss" sources #121Smengerl wants to merge 12 commits into
Conversation
Per review feedback from @j6k4m8 on j6k4m8#121: "content filter" reads as ambiguous (could mean "filter FOR" as much as "filter OUT"), while skip_title_patterns is already unambiguous as-is. Adopted his own suggested name directly - renames the config field, constructor kwarg, schema validator, and every reference in README/example-config/tests. skip_title_patterns is intentionally left unchanged (he confirmed it reads fine as-is). The internal apply_content_filters()/should_skip_title() helper functions in contentfilters.py also keep their names - this is a config-surface rename, not an internal-API one. For future symmetry once accept-mode filters are added (also raised in review): accept_title_patterns and content_accept_filters would match each existing field's own word order (verb-first for title, scope-first for content) rather than forcing both families into an identical shape. 86 tests pass; example-config.json still generates a real PDF end-to-end.
Follow-up to the maintainer's accept/reject nomenclature question on j6k4m8#121: content_skip_filters/skip_title_patterns are denylists, this adds their allowlist counterparts. - content_accept_filters: a list of {"selector": "..."} CSS rules tried in order, keeping only the first matching element's contents instead of the whole parsed tree. Only CSS is supported (discussed on the PR: a regex "accept" would just reduce a story to whatever static phrase the pattern matches, not coherent prose) - falls through to the original html unchanged if nothing matches, so a miss never zeroes out an article. - accept_title_patterns: a list of regexes; only entries matching at least one are kept - e.g. ["amazon", "amzn"] to build a single-company ticker out of an otherwise general business feed. Both apply alongside the existing skip filters (accept narrows first, then skip cleans whatever junk remains within that narrower selection) and are entirely additive - existing configs are unaffected.
...ilters/accept_content_filters Consistent verb-first word order with skip_title_patterns/accept_title_patterns, per j6k4m8's review comment on PR j6k4m8#121 (the two field families were flips of each other: skip_title_patterns but content_skip_filters). Also adds "regex" support to accept_content_filters: unlike the existing "css" type (narrows kept content to one container), a "regex" entry is a whole-story keep/reject gate matched against the fetched article's extracted text - e.g. keeping only articles that actually mention a ticker symbol, the content-level counterpart to accept_title_patterns. Addresses the same comment's second point about regex-based accept filtering. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Smengerl
commented
Aug 3, 2026
Heads up: I just pushed an additional commit to this PR that's a bit of scope creep beyond the original title — apologies in advance.
Alongside the content/title filters, I added two more optional "rss" source fields: min_body_text_length and max_body_text_length, which drop stories whose extracted body's visible text length falls outside a given range (catching failed extractions on the low end, and outlier articles — e.g. I recently had a gigantic hardware review in heise RSS — ballooning the output by more than 200 pages). Also extremely short stories in a feed usually indicate that there is no reasonable content to expect or its another form of add that was not filtered out by the title/content filters.
I bundled it into this PR rather than opening a separate one because it depends on the same contentfilters.py module this PR introduces (visible_text_length lives there, applied at the same point as the other filters, right after skip_content_filters/accept_content_filters). Splitting it out would have meant introducing that module twice.
Happy to pull it back out into its own PR on top of this one if you'd rather review/merge them separately — just let me know.
Picks up the one commit mainline was missing from this branch (a8cf134, "feat: add min/max body-text-length filters to RSS sources") - everything else from feature/rss-content-filters was already merged in previously. mainline had gained equivalent RSSFeedStoryProvider-level support for min_body_text_length/max_body_text_length independently, via a separate branch (feature/rss-body-length-filters, PR #3, commit db4bdf6) that was never opened as an upstream PR. This merge supersedes that with the version that actually is staged for upstream (PR j6k4m8#121), so mainline stays aligned with what happens if all open PRs get accepted - not a parallel, never-to-be-proposed implementation of the same feature. Conflict resolution: - goosepaper/storyprovider/rss.py: the two branches added the identical min_body_text_length/max_body_text_length code in the same spot (already auto-merged); the only real conflict was mainline's own prefer_feed_title parameter (from a different, later branch), kept as-is. - goosepaper/config.py: mainline's comic_type validator (added after the last merge of this branch) conflicted only by proximity with the new min_body_text_length/max_body_text_length validators - kept both. - goosepaper/storyprovider/test_rss.py: the incoming branch's three body-length tests were exact-name duplicates of tests mainline already had (from db4bdf6) - dropped the duplicates, kept mainline's unrelated test_rss_provider_skips_entry_that_raises_without_dropping_the_ whole_feed test that got tangled into the same conflict by proximity. Net new to mainline: goosepaper/config.py's declarative "rss" schema now recognizes min_body_text_length/max_body_text_length (previously only usable by constructing RSSFeedStoryProvider directly in Python, e.g. from the goosepaper-addon wrapper - not via goosepaper's own --config JSON flag), plus matching README.md documentation and an example-config.json usage. Full test suite: 186 passed.
...able Six open PRs (j6k4m8#126-j6k4m8#131, all opened the same day) were never added to the "About this fork" tracking table. j6k4m8#121's row still described its original scope (content_filters + skip_title_patterns) despite the PR having grown substantially since - renamed to skip_content_filters, plus accept_content_filters/accept_title_patterns and min/max_body_text_length.
Smengerl
commented
Aug 9, 2026
Part of a merge-order check across all my open PRs (full breakdown on #124). This PR touches both storyprovider/rss.py and config.py's source-schema block, so it conflicts with the most siblings of anything in the queue: #129, #128, #127 (rss.py) and #133, #132, #124 (config.py) — 6 total. Recommend merging this one last overall, after both the rss.py group and the config.py group land, so it only needs one final rebase instead of repeated ones.
Smengerl
commented
Aug 9, 2026
Heads-up, separate from the sibling-conflict note above: GitHub currently reports this PR as dirty against master — caused by #138 (register_story_provider, merged 2026年08月08日, not one of my PRs) touching test_config.py at the same spot this PR's own additions land. Trivial rebase, unrelated to the sibling-PR conflict clusters.
...cted title (#129) ## Summary - readability's `doc.title()` is unreliable on some sites (e.g. it returns just the site name for every article on some blogs). The RSS feed's own `<title>` is usually accurate. - New `prefer_feed_title` flag, off by default so existing behavior is unchanged; callers who hit this on a specific feed can opt in per source. ## Test plan - [x] `pytest goosepaper/storyprovider/test_rss.py` - 11 passed (9 existing + 2 new: flag on -> feed title wins; flag off/default -> readability title unchanged) - [x] Full suite (`pytest`) - 80 passed ## Merge overlap note Verified by locally merging every pairwise combination of my currently open PRs against `master`. This one produces a mechanical (non-semantic) merge conflict with **#121** only - both add new parameters to `RSSFeedStoryProvider.__init__` at the same point. No overlap with any other currently open PR.
Per review feedback from @j6k4m8 on j6k4m8#121: "content filter" reads as ambiguous (could mean "filter FOR" as much as "filter OUT"), while skip_title_patterns is already unambiguous as-is. Adopted his own suggested name directly - renames the config field, constructor kwarg, schema validator, and every reference in README/example-config/tests. skip_title_patterns is intentionally left unchanged (he confirmed it reads fine as-is). The internal apply_content_filters()/should_skip_title() helper functions in contentfilters.py also keep their names - this is a config-surface rename, not an internal-API one. For future symmetry once accept-mode filters are added (also raised in review): accept_title_patterns and content_accept_filters would match each existing field's own word order (verb-first for title, scope-first for content) rather than forcing both families into an identical shape. 86 tests pass; example-config.json still generates a real PDF end-to-end.
Follow-up to the maintainer's accept/reject nomenclature question on j6k4m8#121: content_skip_filters/skip_title_patterns are denylists, this adds their allowlist counterparts. - content_accept_filters: a list of {"selector": "..."} CSS rules tried in order, keeping only the first matching element's contents instead of the whole parsed tree. Only CSS is supported (discussed on the PR: a regex "accept" would just reduce a story to whatever static phrase the pattern matches, not coherent prose) - falls through to the original html unchanged if nothing matches, so a miss never zeroes out an article. - accept_title_patterns: a list of regexes; only entries matching at least one are kept - e.g. ["amazon", "amzn"] to build a single-company ticker out of an otherwise general business feed. Both apply alongside the existing skip filters (accept narrows first, then skip cleans whatever junk remains within that narrower selection) and are entirely additive - existing configs are unaffected.
...ilters/accept_content_filters Consistent verb-first word order with skip_title_patterns/accept_title_patterns, per j6k4m8's review comment on PR j6k4m8#121 (the two field families were flips of each other: skip_title_patterns but content_skip_filters). Also adds "regex" support to accept_content_filters: unlike the existing "css" type (narrows kept content to one container), a "regex" entry is a whole-story keep/reject gate matched against the fetched article's extracted text - e.g. keeping only articles that actually mention a ticker symbol, the content-level counterpart to accept_title_patterns. Addresses the same comment's second point about regex-based accept filtering. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
680ab0f to
5eb7068
Compare
Adds content_filters (CSS-selector / regex cleanup rules, applied to the extracted article HTML) and skip_title_patterns (regex titles to skip before fetching) as native, optional fields on the "rss" source type. Both are ordinary JSON config - no wrapper code required to use them. - goosepaper/contentfilters.py: apply_content_filters()/should_skip_title(), a regex-then-CSS cleanup pass over fetched article HTML. - goosepaper/storyprovider/rss.py: RSSFeedStoryProvider applies both during get_stories(), skipping matched titles before the network fetch and stripping matched elements/patterns from the extracted body afterward. - goosepaper/config.py, goosepaper/util.py: schema validation and provider wiring for the two new optional "rss" fields. Proven in production against 25+ real-world feeds' ad blocks, cookie banners, and paywall/teaser stubs (heise, t3n, Electrek, netzpolitik, ...).
This branch's original commit added the feature (contentfilters.py, config.py/util.py validation, RSS wiring) but shipped without dedicated test coverage or user-facing docs - fixing both ahead of review: - goosepaper/test_contentfilters.py: unit tests for apply_content_filters (css removal, regex stripping, flags, the regex-before-css ordering) and should_skip_title. - goosepaper/storyprovider/test_rss.py: two integration tests proving RSSFeedStoryProvider actually applies content_filters to a fetched story's body_html and skips entries matching skip_title_patterns before they're counted toward `limit`. - goosepaper/test_config.py: config-schema validation tests - a valid rss source with both fields loads correctly; an unknown filter type, a css filter without `selector`, a regex filter without `pattern`, and an unknown field on a filter object each raise ConfigError with a specific message. - README.md + example-config.json: document both options inline (selector/ pattern/flags shape, ordering, case-insensitive title matching) and add a working example - verified by generating a real PDF from the example config end-to-end.
Per review feedback from @j6k4m8 on j6k4m8#121: "content filter" reads as ambiguous (could mean "filter FOR" as much as "filter OUT"), while skip_title_patterns is already unambiguous as-is. Adopted his own suggested name directly - renames the config field, constructor kwarg, schema validator, and every reference in README/example-config/tests. skip_title_patterns is intentionally left unchanged (he confirmed it reads fine as-is). The internal apply_content_filters()/should_skip_title() helper functions in contentfilters.py also keep their names - this is a config-surface rename, not an internal-API one. For future symmetry once accept-mode filters are added (also raised in review): accept_title_patterns and content_accept_filters would match each existing field's own word order (verb-first for title, scope-first for content) rather than forcing both families into an identical shape. 86 tests pass; example-config.json still generates a real PDF end-to-end.
Follow-up to the maintainer's accept/reject nomenclature question on j6k4m8#121: content_skip_filters/skip_title_patterns are denylists, this adds their allowlist counterparts. - content_accept_filters: a list of {"selector": "..."} CSS rules tried in order, keeping only the first matching element's contents instead of the whole parsed tree. Only CSS is supported (discussed on the PR: a regex "accept" would just reduce a story to whatever static phrase the pattern matches, not coherent prose) - falls through to the original html unchanged if nothing matches, so a miss never zeroes out an article. - accept_title_patterns: a list of regexes; only entries matching at least one are kept - e.g. ["amazon", "amzn"] to build a single-company ticker out of an otherwise general business feed. Both apply alongside the existing skip filters (accept narrows first, then skip cleans whatever junk remains within that narrower selection) and are entirely additive - existing configs are unaffected.
_validate_content_skip_filters used one combined allowed-keys set for
both filter types, so e.g. {"type": "css", "selector": "...",
"pattern": "never used"} passed validation silently - apply_content_filters()
just ignores "pattern" for a "css" entry, so the typo/confusion had no
error to catch it. Key set is now looked up per "type" instead.
Also precisifies README.md's filter documentation: spells out which
keys are required/optional per type for content_skip_filters, and
explicitly contrasts content_accept_filters' simpler always-CSS shape
(no "type" field at all) rather than leaving readers to infer it by
analogy.
Its sibling was always named should_skip_title, mirroring the config field skip_title_patterns exactly - but apply_content_filters kept its pre-rename name when the config field became content_skip_filters (see 42261ee), on the reasoning that it's config-surface vs. internal API. That held while there was only one content-filter function; now that apply_content_accept_filters sits right next to it, the omission reads as accidental rather than intentional - a reader would reasonably guess apply_content_filters is "the general one" rather than immediately seeing "one skips, one accepts". Also renames the four skip-filter tests in test_contentfilters.py that still said generic "filter" (test_css_filter_..., test_regex_filter_...) instead of "skip_filter", for the same symmetry their accept-filter counterparts already had (test_accept_filter_...). No behavior change - pure rename, all 105 tests still pass.
The previous rewrite (d258490) made the filter docs more precise but landed as three dense paragraphs full of "unlike X" cross-references you had to actively untangle - confusing per feedback. Leads with a skip/accept x title/content table showing the four fields' relationship at a glance, then a compact bullet per field instead of comparative prose. Same content, restructured for scanability.
...ilters/accept_content_filters Consistent verb-first word order with skip_title_patterns/accept_title_patterns, per j6k4m8's review comment on PR j6k4m8#121 (the two field families were flips of each other: skip_title_patterns but content_skip_filters). Also adds "regex" support to accept_content_filters: unlike the existing "css" type (narrows kept content to one container), a "regex" entry is a whole-story keep/reject gate matched against the fetched article's extracted text - e.g. keeping only articles that actually mention a ticker symbol, the content-level counterpart to accept_title_patterns. Addresses the same comment's second point about regex-based accept filtering. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
...ng wiring coverage The two separate "accepts_*" tests for skip_content_filters/accept_content_filters mostly duplicated the same generic options passthrough already exercised elsewhere - collapsed into one combined smoke test covering all four filter fields at once. More importantly, nothing previously drove a config through construct_story_providers_from_source_configs() for these four fields, so a typo in util.py's separate allowed-keys copy of the same field names would pass config.py validation but silently drop the option before it reached RSSFeedStoryProvider - added test_construct_story_providers_passes_rss_content_filter_options to close that gap (verified it fails on a deliberately introduced typo before adding it). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
min_body_text_length skips stories whose extracted body is implausibly short (almost always a failed extraction). max_body_text_length adds the inverse: skip stories whose body is implausibly long, e.g. a hardware review with a huge image gallery/spec dump that would otherwise balloon a single RSS entry into the bulk of the whole paper. Both optional and off by default, applied after skip_content_filters/accept_content_filters so the check sees the same body the reader will.
config.py validated these two fields (added alongside the four content- filter fields, but without updating this file to match) - a well-formed "rss" source passed every test_config.py check while the values never reached RSSFeedStoryProvider at all, silently dropped by util.py's own, separate allowed-keys set for what actually gets forwarded to the constructor. Same failure mode the existing test_construct_story_providers_passes_rss_content_filter_options already warns about in its own docstring, just not extended to cover these two newer fields when they were added. Verified: min_body_text_length/max_body_text_length now reach the constructed provider end-to-end via load_paper_config() + construct_story_providers_from_source_configs(), not just Python callers constructing RSSFeedStoryProvider directly. Full suite: 120 passed.
Tags were replaced with a space but newlines/indentation from pretty-printed source HTML were never collapsed, so a short teaser-only extraction from a heavily-indented page could clear min_body_text_length on pure source formatting rather than real content. A rendered page (or a screen reader) collapses that whitespace too, so none of it is actually "visible text".
5eb7068 to
7b58ab3
Compare
Uh oh!
There was an error while loading. Please reload this page.
Why
goosepaper fetches an RSS entry's summary, but for real article content it follows the entry's link and runs
readabilityagainst the full page - and that's where the trouble starts.readabilityextracts "the article", not"the article as the author wrote it": it keeps whatever DOM structure the page's CMS wrapped the actual text in, and most news sites wrap it in a lot.
Cookie/consent banners, gallery/lightbox widgets, "read more"/"show less" teaser toggles, newsletter signup CTAs, related-content boxes, ad slot placeholders - none of that is prose, all of it survives extraction, and all of it printed straight onto the newspaper page with no way to remove it before this PR.
This isn't hypothetical - I built this feature to run my own daily newspaper (~25 RSS feeds) and needed it from day one to keep the output readable. Some concrete examples from real feeds, all currently running in production:
<details>, gift-article/opt-in custom elements, "Mehr/Weniger anzeigen" toggle text{"type": "css", "selector": "div.Gallery"},div[class*='ad-mobile-group'],details.notice-banner,a-gift,a-opt-in, plus{"type": "regex", "pattern": "Mehr anzeigen"}/"Weniger anzeigen"skip_title_patterns: ["^anzeige:", "^heise\\+ \|", "^heise-angebot:"]div[data-consent-service-id],div.c-suggestNews-container,a.t-newsletter-singleview-top-ctacssselectors, plus{"type": "regex", "pattern": "(considering going solar|FTC: We use income earning).*", "flags": "s"}(thesflag so.also matches the newlines in that trailing block)dividentified only by adata-moduleattributediv.RelatedContent-relatedContent,[data-module='mps-slot']div.netzpolitik-ctaa.gf-pilla[href^='#']:has(svg)(also proves the CSS:has()selector works through this pipeline's selector engine -soupsieve)div#cmp-contentskip_title_patterns: ["^view photos of"]Doing this as a wrapper around goosepaper isn't really possible: the
cleanup has to run on the exact HTML
RSSFeedStoryProvideralreadyextracted via
readability, insideget_stories()- an external layer hasno hook into that step without re-implementing the whole fetch/extract
pipeline itself. So it belongs here, as new optional, plain-JSON fields
on the
"rss"source type, no custom code required to use them.Grown since first opened: this PR started out with just a
content_filtersdenylist andskip_title_patterns. Review/production use surfaced the need for the allowlist direction too (keep only what matches, not just drop what doesn't), socontent_filterswas renamed toskip_content_filtersonce its allowlist counterpartaccept_content_filtersexisted - "skip" alongside a bare "content_filters" read as if it were the only direction.min_body_text_length/max_body_text_lengthwere added for the same production feeds, to catch extraction failures and outlier-length articles that no selector/regex rule targets. The table above and the rest of this description reflect the current, full scope.What
RSS sources can filter along two independent axes - what to match (title or fetched article content) and which direction (skip = denylist, accept = allowlist):
skip_title_patternsaccept_title_patternsskip_content_filtersaccept_content_filtersskip_title_patterns/accept_title_patterns- flat list of regexes, matched case-insensitively against the entry title, before it's even fetched.skip: a match drops the entry (e.g.["^anzeige:", "^sponsored"]to drop sponsored posts).accept: if non-empty, only matches are kept (e.g.["amazon", "amzn"]to build a single-company news ticker out of an otherwise general feed).skip_content_filters- list of{"type": "css", "selector": "..."}(deletes matching elements, e.g. ad blocks or cookie banners) or{"type": "regex", "pattern": "...", "flags": "i"}(strips matching text;flagsoptional, any ofi/s/m/x) rules.typedecides which other keys are valid - acssentry can't carrypattern/flags, aregexentry can't carryselector. Regex rules run first, then CSS rules, regardless of list order.accept_content_filters- list of{"type": "css", "selector": "..."}or{"type": "regex", "pattern": "...", "flags": "i"}rules, tried/applied independently of each other.css: the first selector that matches wins and the article is replaced with just that element's contents - useful whenreadability's own extraction misses and you know exactly which container holds the real content; no match leaves the article unchanged.regex: a whole-story gate rather than a transform - matched against the fetched article's text (not raw markup), so a story is kept only if it matches at least oneregexfilter; acssfilter can't sensibly gate this way, since a miss should leave the article as-is rather than drop it.min_body_text_length/max_body_text_length- optional, applied after the content filters above; drop stories whose extracted body's visible text length falls outside that range.mincatches a failed extraction (a near-empty body);maxcatches an implausibly long body (e.g. a hardware review with a huge photo gallery/spec dump) that would otherwise balloon a single entry into the bulk of the whole paper.Reproduction
A feed returning two entries - one with a sponsored-post title, both
sharing the same messy article HTML (an ad block, a cookie banner, and a
trailing "Mehr anzeigen" link, modeled directly on the heise example above):
Before (master): no way to express this in config at all -
none of the fields above are recognized parameters. Both
entries come through untouched, junk included:
2 stories returned:
--- Anzeige: Sponsored post you'd rather skip ---
Real article content starts here.
More real content in the middle.
Mehr anzeigen
--- A real headline about something interesting ---
Real article content starts here.
More real content in the middle.
Mehr anzeigen
After (this branch), with:
1 story returned:
--- A real headline about something interesting ---
Real article content starts here.
More real content in the middle.
The sponsored entry is gone entirely, and the ad block/cookie banner/tracking text are stripped from the survivor. (The wrapper and the leftover empty
are BeautifulSoup's own re-serialization once at least one CSS filter runs - cosmetically harmless, WeasyPrint renders it the same either way.)Implementation
goosepaper/contentfilters.py(new):apply_skip_content_filters()/apply_accept_content_filters()/should_skip_title()/should_accept_title()/should_accept_content()/visible_text_length()- the actual filtering logic, framework-independent.goosepaper/storyprovider/rss.py:RSSFeedStoryProviderapplies title-level filters first thing in itsget_stories()loop (before any fetch), then content-level filters and the body-length gate on the extractedbody_htmlright after fetching - regardless of whichbody_sourcemode produced that HTML.goosepaper/config.py,goosepaper/util.py: schema validation and provider wiring for all six new optional"rss"fields.Testing
test_contentfilters.py(new): unit tests for every function above (css/regex removal and gating, flags, ordering, body-length thresholds).test_rss.py: integration tests provingRSSFeedStoryProvideractually applies each new field to a fetched story, and that title-level skip/accept happens before an entry counts towardlimit.test_config.py: schema validation for all six fields, including the error message for each malformed case (unknown filter type, missing selector/pattern, cross-type fields, unknown fields).example-config.json(updated with working examples of every field) generates a real PDF end-to-end.Scope
Additive only - all six fields are optional, existing "rss" sources without them are completely unaffected.
Merge overlap note
Verified by locally merging every pairwise combination of my currently open PRs against
master. This one produces a real but purely mechanical (non-semantic) merge conflict with #123, #124, #127, #128, #129, #132, #133 - all add entries to one of the same shared registries this PR also touches (config.py's per-field validator dict,util.py's provider dispatch dict, orrss.py'sRSSFeedStoryProvider.__init__/get_stories). In every case checked, the conflicting insertions are independent (different field/provider names, no logic overlap) - resolving means keeping both sides, not choosing between them.