21
53
Fork
You've already forked sciop
23

WIP: Migration for recalculating torrent size #295

Draft
Astatide wants to merge 1 commit from astatide/ISSUE-287-torrent-size-migration into main
pull from: astatide/ISSUE-287-torrent-size-migration
merge into: Safeguarding:main
Safeguarding:main
Safeguarding:dataset-scopes
Safeguarding:filter-items
Safeguarding:cli-trackers
Safeguarding:mkdocs-icon-link
Safeguarding:create_time_index
Safeguarding:random-quote-config
Safeguarding:faster-counter
Safeguarding:bugfix-config-defaults
Safeguarding:spotlight-svg-convert
Safeguarding:crawlercatcher
Safeguarding:performance-logging
Safeguarding:allow-replacing-identical-ih-torrents
Safeguarding:disable-animation
Safeguarding:rss-feed-cache-2
Safeguarding:mark-ui
Safeguarding:src-icon
Safeguarding:unapprove
Safeguarding:docs-usage
Safeguarding:footer-md
Safeguarding:rssfeedadditions
Safeguarding:magnet-link-when-collapsed
Safeguarding:hit-counter
Safeguarding:css-wrapping
Safeguarding:config-yaml
Safeguarding:obscure-email
Safeguarding:aboutpage
Safeguarding:more-torrent-feeds
Safeguarding:bugfix-nested-sort
Safeguarding:docs-griffe-pydantic
Safeguarding:rsscache
Safeguarding:bugfix-sort-numbers
Safeguarding:sort-lists
Safeguarding:bump-limits
Safeguarding:magnet-links
Safeguarding:ttl-url-fix
Safeguarding:sort-dataset-parts
Safeguarding:nixify
Safeguarding:docs-no-build-in-prod
Safeguarding:docs-api-data-structures
Safeguarding:docs-webscraping-cleaning
Safeguarding:perf
Safeguarding:logging-400s
Safeguarding:bugfix-search-syntax
Safeguarding:hotfix-version-uq-bug
Safeguarding:tests-playwright
Safeguarding:tests-warnings-cleanup
Safeguarding:docs-bittorrent
Safeguarding:contact
Safeguarding:request-scope
Safeguarding:ignore-padfiles
Safeguarding:datetime-inputs
Safeguarding:markdown-blockquotes
Safeguarding:lifespan-run-once
Safeguarding:http-scrape
Safeguarding:docs-rss
Safeguarding:continuous-deployment
Safeguarding:dataset-parts-description-fix
Safeguarding:the-most-minor-fix-ever
Safeguarding:css-uploads-table-width
Safeguarding:bugfix-response-type
Safeguarding:site-stats
Safeguarding:patch-null-identifier-uri
Safeguarding:ci-python-matrix
Safeguarding:bugfix-torf-max-size
Safeguarding:docs-index-links
Safeguarding:db-case-insensitive-username
Safeguarding:frontend-limits
Safeguarding:docs-clients
Safeguarding:bugfig-watch-dirs
Safeguarding:docs-bassline
Safeguarding:db-fk-updates
Safeguarding:bittorrent-source-type
Safeguarding:frontpage-beta
Safeguarding:cli-generate
Safeguarding:bugfix-duplicate-torrents
Safeguarding:bugfix-form-bools
Safeguarding:bugfix-pk-migration
Safeguarding:rebuild-search-idx
Safeguarding:cli
Safeguarding:editing
Safeguarding:rm-mimetype-checking
Safeguarding:schoeneh-patch-1
Safeguarding:rm-model-formitem-followup
Safeguarding:bugfix-last-seen-at
Safeguarding:bugfix-upload-torrent
Safeguarding:ci-solaris-runner
Safeguarding:markdown-touchups
Safeguarding:torrent-scrape-servie
Safeguarding:astatide/apscheduler
Safeguarding:error-handling-non-torrent-uploads
Safeguarding:filter-user-page
Safeguarding:frontpage-message
Safeguarding:db-migrations-actually
Safeguarding:delete-buttons
Safeguarding:upload-without-scope
Safeguarding:dont-delete-things-self
Safeguarding:minor-patch
Safeguarding:htmx-source-map
Safeguarding:markdown-descriptions
Safeguarding:front-page-text
Safeguarding:astatide/upload-db-seeders-leechers-update
Safeguarding:astatide/torrent-stat

This (ideally) addresses #287

This pulls in the file list from the torrent_file table, calculates the sum of the sizes on the disk, then updates the value. I'm not sure whether this should include checks for empty files, whether these are the appropriate libraries, whether it will work, hence why it's a WIP!

This (ideally) addresses https://codeberg.org/Safeguarding/sciop/issues/287 This pulls in the file list from the torrent_file table, calculates the sum of the sizes on the disk, then updates the value. I'm not sure whether this should include checks for empty files, whether these are the appropriate libraries, whether it will _work_, hence why it's a WIP!
Add in initial migration for recalculating torrent size
Some checks failed
lint / lint (pull_request) Failing after 12s
tests / tests-playwright-python3.11 (pull_request) Successful in 47s
tests / tests-playwright-python3.12 (pull_request) Successful in 50s
tests / tests-playwright-python3.13 (pull_request) Successful in 49s
tests / tests-python3.11 (pull_request) Failing after 1m35s
tests / tests-python3.12 (pull_request) Failing after 1m38s
tests / tests-python3.13 (pull_request) Failing after 1m37s
8497aad642
@ -0,0 +35,4 @@
sa.select(table.c.id, table.c.files, table.c.size)
).fetchall()
for id_, _files in results:
Author
Owner
Copy link

super consistent naming here auds (prepending vs. appending an underscore)

super consistent naming here auds (prepending vs. appending an underscore)
@ -0,0 +36,4 @@
).fetchall()
for id_, _files in results:
files = [Path.joinpath(config.torrent_dir, file).stat().st_size for file in _files]
Author
Owner
Copy link

this should not be a one liner

this should not be a one liner
@ -0,0 +27,4 @@
sa.MetaData(),
sa.Column("id", sa.Integer(), key="id"),
sa.Column("files", sa.String(), key="files"),
# sa.Column("total_size", sa.Integer(), key="size"),
Author
Owner
Copy link

Not the most experienced SQL cat so I wasn't sure whether my selection needed to include the existing field or not, so I uhh wrote it and commented it out

Not the _most_ experienced SQL cat so I wasn't sure whether my selection needed to include the existing field or not, so I uhh wrote it and commented it out
@ -0,0 +25,4 @@
table = sa.Table(
"torrent_files",
sa.MetaData(),
sa.Column("id", sa.Integer(), key="id"),
Author
Owner
Copy link

I'm also not 100% sure whether the key field is necessary or not; I'm assuming it's to facilitate aliasing in some manner?

I'm also not 100% sure whether the key field is necessary or not; I'm assuming it's to facilitate aliasing in some manner?
Some checks failed
lint / lint (pull_request) Failing after 12s
Required
Details
tests / tests-playwright-python3.11 (pull_request) Successful in 47s
tests / tests-playwright-python3.12 (pull_request) Successful in 50s
tests / tests-playwright-python3.13 (pull_request) Successful in 49s
tests / tests-python3.11 (pull_request) Failing after 1m35s
tests / tests-python3.12 (pull_request) Failing after 1m38s
tests / tests-python3.13 (pull_request) Failing after 1m37s
Required
Details
This pull request is marked as a work in progress.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin astatide/ISSUE-287-torrent-size-migration:astatide/ISSUE-287-torrent-size-migration
git switch astatide/ISSUE-287-torrent-size-migration
Sign in to join this conversation.
No reviewers
Labels
Clear labels
accessibility
backend
bittorrent
component/cli
docs
feature/feeds
feature/moderation
feature/search
feature/services
Scheduled services run by the apscheduler runner
frontend
help wanted
Pitches for ideas for which a PR would be welcome <3. Everything is always help wanted, but these are things that would be discrete features that someone could take and implement on their own
perf
security
size
large
 changes to many items, hundreds-thousands of lines, doable in >2 days
size
medium
 changes to a handful of items, ~dozens to hundreds of lines, doable in ~1-2 days
size
small
 changes to a single or small number of items, ~a dozen or two lines, doable in ~1-2 hours
size
tiny
 a line or two
tests
Timeline
Long
 months, years
Timeline
Medium
 weeks, up to a month
Timeline
Short
 Hours, days, up to a week
ui/ux
zzz-certified-good-issue-or-pr-number
Compat/Breaking
Breaking change that won't be backward compatible
Kind/Bug
Something is not working
Kind/Documentation
Documentation changes
Kind/Enhancement
Improve existing functionality
Kind/Feature
New functionality
Kind/Security
This is security issue
Kind/Tech Debt
Kind/Testing
Issue or pull request related to testing
Priority
Critical
 The priority is critical
Priority
High
 The priority is high
Priority
Low
 The priority is low
Priority
Medium
 The priority is medium
Reviewed
Confirmed
 Issue has been confirmed
Reviewed
Duplicate
 This issue or pull request already exists
Reviewed
Invalid
 Invalid issue
Reviewed
Won't Fix
 This issue won't be fixed
Status
Abandoned
 Somebody has started to work on this but abandoned work
Status
Blocked
 Something is blocking this issue or pull request
Status
Need More Info
 Feedback is required to reproduce issue or to continue work
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Safeguarding/sciop!295
Reference in a new issue
Safeguarding/sciop
No description provided.
Delete branch "astatide/ISSUE-287-torrent-size-migration"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?