21
53
Fork
You've already forked sciop
23

[backend] Caching feeds #8

Closed
opened 2025年02月04日 10:22:22 +01:00 by sneakers-the-rat · 5 comments

should probably cache them at least over some interval, since those will be the primary traffic drivers likely

should probably cache them at least over some interval, since those will be the primary traffic drivers likely
Contributor
Copy link

We can go for (削除) redis (削除ここまで) valkey or memcached for that, and then I think it would be great we start providing a docker compose if it needs to spin up a few different services (valkey for caching and possibly for rate limiting is a great solution I think).

What do you think about that?

We can go for ~~redis~~ valkey or memcached for that, and then I think it would be great we start providing a docker compose if it needs to spin up a few different services (valkey for caching and possibly for rate limiting is a great solution I think). What do you think about that?
Author
Owner
Copy link

I am a little hesitant to make docker the default way to deploy this, which is sorta what happens when you start bundling multiple services together, because that really raises the complexity and resource requirements of rehosting. Like the fastapi/htmx design rather than standard frontend/backend is intended to be very very deployable and resource light.

I would like to know if there is some caching service we can spawn from within python in a reasonable way, but I know redis/valkey is the standard and have used it to good effect in the past, so I also don't want my docker aversion to stand in the way of this getting done and invite proposals on what an impl might look like here

I am a *little* hesitant to make docker the default way to deploy this, which is sorta what happens when you start bundling multiple services together, because that really raises the complexity and resource requirements of rehosting. Like the fastapi/htmx design rather than standard frontend/backend is intended to be very very deployable and resource light. I would like to know if there is some caching service we can spawn from within python in a reasonable way, but I know redis/valkey is the standard and have used it to good effect in the past, so I also don't want my docker aversion to stand in the way of this getting done and invite proposals on what an impl might look like here
Contributor
Copy link

It can be in memory cache but it might grow a bit bigger in memory usage.
A first iteration can be with fastapi-cache in memory and if later there's a need for more, changing the caching backend is easy

It can be in memory cache but it might grow a bit bigger in memory usage. A first iteration can be with [fastapi-cache](https://github.com/long2ice/fastapi-cache) in memory and if later there's a need for more, changing the caching backend is easy
Author
Owner
Copy link

hmm yeah, i'll keep my mind open and defer to running code over my aversion to docker in any case. it seems like we'll want to cache at the query level and at the response level esp as we move towards effectively implementing a tracker here. still trying to just get a MVP so i am trying to stay a little perf agnostic right now - get it working, then make it fast, but i admit i am esp vulnerable to the temptation to preoptimize

hmm yeah, i'll keep my mind open and defer to running code over my aversion to docker in any case. it seems like we'll want to cache at the query level and at the response level esp as we move towards effectively implementing a tracker here. still trying to just get a MVP so i am trying to stay a little perf agnostic right now - get it working, then make it fast, but i admit i am esp vulnerable to the temptation to preoptimize
Author
Owner
Copy link

done awhile ago!

done awhile ago!
Sign in to join this conversation.
No Branch/Tag specified
main
dataset-scopes
filter-items
cli-trackers
mkdocs-icon-link
create_time_index
random-quote-config
faster-counter
bugfix-config-defaults
spotlight-svg-convert
crawlercatcher
performance-logging
allow-replacing-identical-ih-torrents
disable-animation
rss-feed-cache-2
mark-ui
src-icon
unapprove
docs-usage
footer-md
rssfeedadditions
magnet-link-when-collapsed
hit-counter
css-wrapping
config-yaml
obscure-email
aboutpage
more-torrent-feeds
bugfix-nested-sort
docs-griffe-pydantic
rsscache
bugfix-sort-numbers
sort-lists
bump-limits
magnet-links
ttl-url-fix
sort-dataset-parts
nixify
docs-no-build-in-prod
docs-api-data-structures
docs-webscraping-cleaning
perf
logging-400s
astatide/ISSUE-287-torrent-size-migration
bugfix-search-syntax
hotfix-version-uq-bug
tests-playwright
tests-warnings-cleanup
docs-bittorrent
contact
request-scope
ignore-padfiles
datetime-inputs
markdown-blockquotes
lifespan-run-once
http-scrape
docs-rss
continuous-deployment
dataset-parts-description-fix
the-most-minor-fix-ever
css-uploads-table-width
bugfix-response-type
site-stats
patch-null-identifier-uri
ci-python-matrix
bugfix-torf-max-size
docs-index-links
db-case-insensitive-username
frontend-limits
docs-clients
bugfig-watch-dirs
docs-bassline
db-fk-updates
bittorrent-source-type
frontpage-beta
cli-generate
bugfix-duplicate-torrents
bugfix-form-bools
bugfix-pk-migration
rebuild-search-idx
cli
editing
rm-mimetype-checking
schoeneh-patch-1
rm-model-formitem-followup
bugfix-last-seen-at
bugfix-upload-torrent
ci-solaris-runner
markdown-touchups
torrent-scrape-servie
astatide/apscheduler
error-handling-non-torrent-uploads
filter-user-page
frontpage-message
db-migrations-actually
delete-buttons
upload-without-scope
dont-delete-things-self
minor-patch
htmx-source-map
markdown-descriptions
front-page-text
astatide/upload-db-seeders-leechers-update
astatide/torrent-stat
No results found.
Labels
Clear labels
accessibility
backend
bittorrent
component/cli
docs
feature/feeds
feature/moderation
feature/search
feature/services
Scheduled services run by the apscheduler runner
frontend
help wanted
Pitches for ideas for which a PR would be welcome <3. Everything is always help wanted, but these are things that would be discrete features that someone could take and implement on their own
perf
security
size
large
changes to many items, hundreds-thousands of lines, doable in >2 days
size
medium
changes to a handful of items, ~dozens to hundreds of lines, doable in ~1-2 days
size
small
changes to a single or small number of items, ~a dozen or two lines, doable in ~1-2 hours
size
tiny
a line or two
tests
Timeline
Long
months, years
Timeline
Medium
weeks, up to a month
Timeline
Short
Hours, days, up to a week
ui/ux
zzz-certified-good-issue-or-pr-number
Compat/Breaking
Breaking change that won't be backward compatible
Kind/Bug
Something is not working
Kind/Documentation
Documentation changes
Kind/Enhancement
Improve existing functionality
Kind/Feature
New functionality
Kind/Security
This is security issue
Kind/Tech Debt
Kind/Testing
Issue or pull request related to testing
Priority
Critical
The priority is critical
Priority
High
The priority is high
Priority
Low
The priority is low
Priority
Medium
The priority is medium
Reviewed
Confirmed
Issue has been confirmed
Reviewed
Duplicate
This issue or pull request already exists
Reviewed
Invalid
Invalid issue
Reviewed
Won't Fix
This issue won't be fixed
Status
Abandoned
Somebody has started to work on this but abandoned work
Status
Blocked
Something is blocking this issue or pull request
Status
Need More Info
Feedback is required to reproduce issue or to continue work
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Safeguarding/sciop#8
Reference in a new issue
Safeguarding/sciop
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?