WIP - scopes refactor for multiple dataset owners #456

@sneakers-the-rat so my main questions are:

1. current implementation has 3 scopes for datasets: edit, permissions (for granting/revoking permissions for other users), delete ... any others I should add to the mix?
2. current implementation replaces dataset.account == current_account with current_account.has_scope for editable/removable_by ... do we want to retain the former as well? in other words, do we want to allow a dataset's creator to have their scopes revoked? (migration creates scopes for dataset creators, so nobody will lose edit/delete access to datasets they created if we keep it as is)
3. related to 2, in those editable/removable_by checks, I'm looking for the appropriate scope whenever hasattr(self, "dataset_id"), which provides a bit of hierarchy for models which have dataset_id as a foreign key (dataset parts, uploads, etc), so, for example, users with the edit scope for a dataset can also edit that dataset's uploads. for uploads though, I did retain the self.account == current_account so uploaders can still edit/remove their uploads for datasets that they don't have scopes for. not really a question, but wanna make sure that all makes sense.

other than that, the gist is...

• a new AccountDatasetScopeLinks table, pmuch AccountScopeLinks but with a dataset_id foreign key... i was originally thinking of turning AccountScopeLinks into a more general purpose ScopeLinks with an integer primary key and a unique constraint across account/dataset/upload ID foreign keys, which would each be nullable. but after getting further along it seemed like it'd be a bit over-engineering for a few reasons. and the main thing we're looking for here is multiple dataset owners, so went with this for simplicity. but might be worth doing that if we want scopes to be applied to more models, and it wouldn't be too much to refactor and maybe do part of it as a mixin.
• a new "collaborators" field in the dataset form, with tokens that look similar to the admin page's account scope rows. but unlike the account scopes which make database changes whenever you click on the scope buttons, changes only happen when you submit the form. but I also included a couple API routes for granting/revoking individual dataset scopes to users.
• regular users (non-review/admin/root) can only modify permissions that they've already been granted, and can't modify their own permissions (e.g., this screenshot was taken as the rando user)
  Screenshot From 2025年09月27日 15-49-47

other things in this pr that aren't related to multiple dataset owners/scopes stuff, but I included anyways:

• updated tags autocomplete to exclude already selected tags... also prevented adding tokens with <2 chars on the frontend, since that's the min for the model
• made it so all form fields have the magenta border, which was only showing for a few fields
• added a block to pyproject.toml so my editor won't complain about not finding packages and whatnot
• datasets without a description were displaying "None" because that was the value of the field, so I changed that to display "No description provided"

gotta write tests, but otherwise feel free to review/request changes!

making a couple comments just reading your comment before reading the code since i have a few minutes before monsterdon starts:

permissions (for granting/revoking permissions for other users)

this seems like it could be a little tricky, esp. as an othogonal scope to adding/removing other users from the ownership set (e.g. could i add someone as an owner and then not be able to give them any permissions?). ability to edit permissions is currently done by admin scope which i sort of think is different in kind than other permissions, which are what kind of things you can do, vs. what kind of account you are, so not my favorite, but it's also sort of important for permissions to be able to designate someone as the "primary" on something (esp. while the edit histories are on the fritz so we can't just do nondestructive rollbacks). not rly sure, will think more after reading code.

do we want to allow a dataset's creator to have their scopes revoked?

no i don't think so, unless they give it away or have it revoked by a moderation action

I'm looking for the appropriate scope whenever hasattr(self, "dataset_id")

i'll take a closer look when i read the code, but from the description i'd be a little concerned about having this in the root method implementation, since the odds we forget it's in there and have some other model that has dataset_id in it and give ourselves off target effects seems somewhat high, but lemme read first.

turning AccountScopeLinks into a more general purpose ScopeLinks

I think we'll eventually need to get here, but we may want to move this to graph DB land where this would be easier to do, and to generalize this in sql land should be relatively straightforward to convert the special AccountDatasetScopeLinks table to a more general form. totally reasonable call.

unlike the account scopes which make database changes whenever you click on the scope buttons, changes only happen when you submit the form

makes perfect sense

made it so all form fields have the magenta border,

this was intended to indicate required/not required (purple border == required), but let me see the changes

otherwise looks good! love the frontend screenshot. will give an actual review l8r

@sneakers-the-rat wrote in #456 (comment):

permissions (for granting/revoking permissions for other users)

this seems like it could be a little tricky, esp. as an othogonal scope to adding/removing other users from the ownership set (e.g. could i add someone as an owner and then not be able to give them any permissions?). ability to edit permissions is currently done by admin scope which i sort of think is different in kind than other permissions, which are what kind of things you can do, vs. what kind of account you are, so not my favorite, but it's also sort of important for permissions to be able to designate someone as the "primary" on something (esp. while the edit histories are on the fritz so we can't just do nondestructive rollbacks). not rly sure, will think more after reading code.

ah, so that's a thing i didn't mention. I put the existing scopes (admin, upload, etc) into a new string enum called AccountScopes and dataset scopes onto one called ItemScopes. the Scopes enum that existed before still exists and combines both, but that's pmuch just used for Scopes table stuff now. for displaying scopes on the admin page, it pulls from AccountScopes, and from ItemScopes in the dataset form. this also is reflected in other places like, for example, the RequireValidScope route dep is now RequireValidAccountScope and RequireValidItemScope for the respective grant/revoke account/dataset scopes routes.

this does complicate the way we think about scopes, but it seemed to me that the functionalities we want to provide for multiple dataset owners (edit, delete, etc) don't map onto existing instance-wide account scopes, and vice versa. and even if we want to just use a single scope that grants full ownership to each "collaborator", trying to use existing scopes for both purposes (e.g., { scope: "admin", dataset: {...}, account: {...} }) is risky wrt future work both because it muddies the water for that scope, and because it opens the door to forgetting about the dataset use case for the scope and introducing bugs where users get abilities they shouldn't have. so separation of concerns wrt scope values seems like a good idea here to me.

for clarity, given the above, you still have to be an admin to modify instance-wide account scopes (upload, submit, etc), but any user can modify users' scopes for a given dataset if they have the permissions scope for that dataset.

also don't think i said that the permissions scope includes the edit scope, similar to how the review/admin/root scopes include the upload and submit scopes. because the "collaborators" scope field is in the dataset form and a permissions scope without an edit scope would be useless.

i also think granular scopes for datasets will be useful for future work, as opposed to granting full ownership to each collaborator. e.g., since we want to enable hidden/dark datasets, we could add a "visible" scope that would allow certain users to see and upload to a dataset without being able to edit or delete it.

do we want to allow a dataset's creator to have their scopes revoked?

no i don't think so, unless they give it away or have it revoked by a moderation action

righto, easy to add that back. the current implementation basically assumes that there's no hierarchy among dataset owners (ie, anyone with the permissions scope can revoke any other user's scopes for that dataset). which yeah, could enable bad actors who gets the right permissions to mess up a dataset, esp without being able to do rollbacks, but the idea is also that it's trusted people (like people affiliated with an org) getting permissions, right? i think a teams feature would also be a useful future feature, so users could create teams and grant permissions to the entire team, and removing someone from the team would result in loss of privs for datasets that the team has scopes for.

I'm looking for the appropriate scope whenever hasattr(self, "dataset_id")

i'll take a closer look when i read the code, but from the description i'd be a little concerned about having this in the root method implementation, since the odds we forget it's in there and have some other model that has dataset_id in it and give ourselves off target effects seems somewhat high, but lemme read first.

def something i was concerned about too... could just override those methods for the dataset class to avoid changing the root method... and could still achieve the hierarchy i described (where dataset owners can edit the dataset's uploads), if desired, by also overriding the methods in classes where we want that. i'll go ahead and do that since it'd be more future proof.

turning AccountScopeLinks into a more general purpose ScopeLinks

I think we'll eventually need to get here, but we may want to move this to graph DB land where this would be easier to do, and to generalize this in sql land should be relatively straightforward to convert the special AccountDatasetScopeLinks table to a more general form. totally reasonable call.

ya know what, i'll go ahead and try out the more general approach and see if my concerns were even valid

made it so all form fields have the magenta border,

this was intended to indicate required/not required (purple border == required), but let me see the changes

ah ok, i actually thought that was the case at first but made the change bc i found a discrepancy between required fields and fields with magenta borders at one point. so i'll remove that change and see if i can i replicate what i'd seen (was a while ago, forgot exactly what it was) in case there's a different fix needed to bring the borders and required into alignment

book-length response, sorryyyyy 😜

one more thing - do we display collaborator usernames on the dataset page? should probably do that.

sorry just saw your response after i submitted review. i think i responded to some things after having read the code in comments, but responding to more specific things here:

think a teams feature would also be a useful future feature, so users could create teams and grant permissions to the entire team, and removing someone from the team would result in loss of privs for datasets that the team has scopes for.

agreed that teams/groups should be the eventual form of this. i like having both, where i think teams/groups should basically be addressable as if it was a single account, so e.g. one could have one-off collaborators on some items in addition to it belonging to a group, etc.

i'll go ahead and try out the more general approach and see if my concerns were even valid

if you want to! this is already plenty, so if it's a substantial amount of work to do, then i wouldn't worry about it.

@sneakers-the-rat getting back on this finally. writing tests still, but any idea why black is requesting reformatting in the failed lint check for the last commit? locally black is saying those files are good :P

Oh ya that got me too. Black 26 changed string formatting to make it more compact. I liked the new style so went with that rather than disabling/pinning the version in #481

Also changed the tests so they use the lockfile instead of the most recent versions of packages since this is "an application" after all. If you rebase/merge main and reinstall from the lockfile you should be good

ah, yeah shoulda thought to try that lol. thanks!

@sneakers-the-rat how do we want to handle "reported account"/report actions for multi-owner datasets?

this method is expecting the report target to have an account link if the target isn't an account itself. I removed the datasets.account_id field because it wasn't being used for anything after I switched the submitted by label on dataset pages to be maintained by and display everyone with scopes for the dataset.

I could add it back for reporting purposes, but the person who submitted the reported dataset might not be the person who violated some site rule if the dataset has multiple owners.

OTOH, if we do want to remove datasets.account_id, then for ReportAction.suspend_remove we have a choice to make - does it remove all datasets the account has scopes for even if some of the datasets have other owners? Do we just skip auto-removing datasets altogether? Do we only auto-remove datasets where the suspended account was the sole owner? For the last option, malicious users would be able to give dataset scopes to random users to prevent the datasets from being automatically removed, but maybe we could just display a table of the account's datasets that have other owners when the person handling the report clicks the "suspend remove" button so they can select which datasets to keep and which to remove.

very good questions:

• 1. I think if datasets are truly multi-owner they should behave like it and be modeled like it, so yes i think we shouldn't have a separate field just for reporting purposes. I think the basic behavior we want is "if a dataset with multiple owners is reported and we want to take action against the account as well as the dataset, then moderators should be able to select which of the accounts action is being taken against"
• 2. I think for removing, by default we should not remove datasets that are owned by multiple accounts when one of them is suspended. maybe only when all of the accounts are suspended. I like your suggestion to be able to select/deselect datasets to remove, let's go with that. would it be a lot of work to prefill that form with like "all the datasets that are solely owned by an account" already checked?
• 3. It's a good point re: abuse and randomly adding other people as co-maintainers. In general i think we want to not have it be possible to people to add other people to things, so ideally that would be a request flow: either the current owner requests to add someone to a dataset and they approve/deny, or someone else who wants to become an owner can request it and the current maintainers can approve/deny. That might be a decent bit of work and this PR is already a ton of work, so we might want to come back to that in another PR.

that all makes sense to me!

re: prefilling forms with solely-owned datasets, that should be easy. will do!

and yeah, a request flow def sounds like a good idea... and would be good to already have implemented for other stuff like teams. I'll think on that a bit, but yeah, it might deserve its own PR if it should work for different resources like that.