WIP: feat: Add Gists #6125

This PR adds Gist support to Forgejo. Gists are small Git Repos. They are used to share Codesnipets.

Implementation:

• Each Gist has it's own random UUID (e.g. 7f4d4b2b). They can be accessed using the /gists/{gistuuid} route.
• You can have multiple Gists with the same Name (e.g. Log)
• Gist Repos can't contain binary files
• Gist Repos can't contain directories
• Gist are only allowed to have a main branch.
• Gist can only have a Owner, which has write permission. There is no way to give others Permissions like with Repos.
• Gist don't have Issues or Pull Requests (Comments might be added in a follow up PR)
• A Gist has one of 3 visibility levels:
  • Public: The Gist can be seen be anyone
  • Hidden: The Gist can be seen by anyone but does not appear in the Search
  • Private: The Gist can only be seen be the Owner
• Gists are saved in a diffrent directory than normal Repos
• Gists are saved independently to Repos in the Database

Screenshots:

Why W.I.P.

• This PR is currently missing tests
• Some Bugs are still there
• Some polish is needed

The Feature itself is fully finished and can be reviewed.

Implements #1069

Checklist

The contributor guide contains information that will be helpful to first time contributors. There also are a few conditions for merging Pull Requests in Forgejo repositories. You are also welcome to join the Forgejo development chatroom.

Tests

• I added test coverage for Go changes...
  • in their respective *_test.go for unit tests.
  • in the tests/integration directory if it involves interactions with a live Forgejo server.
• I added test coverage for JavaScript changes...
  • in web_src/js/*.test.js if it can be unit tested.
  • in tests/e2e/*.test.e2e.js if it requires interactions with a live Forgejo server (see also the developer guide for JavaScript testing).

Documentation

• I created a pull request to the documentation to explain to Forgejo users how to use this change.
• I did not document these changes and I do not expect someone else to do it.

Release notes

• I do not want this change to show in the release notes.
• I want the title to show in the release notes with a link to this pull request.
• I want the content of the release-notes/<pull request number>.md to be be used for the release notes instead of the title.

I think this needs to be ENABLED=false by default, at lease for the first time it appears in a major release. It should be advertised as experimental and subject to breaking changes with no warning. The UI and UX for such a new feature is very difficult to get right the first time and you don't want to be stuck with people who rely on it thinking it is final... before it actually is final.

For these reasons, the UI must display a warning so it serves as a reminder.

Does that sound reasonable?

There is a longer discussion related to this suggestion at forgejo/discussions#246

Ooh, exciting feature! I pulled down the branch to play around and have some feedback from experimental testing:

• When a Gist is created in the UI with a filename that contains a path separator (in this case, dir/path.rs) , a 500 Internal Server error occurs; I assume this isn't intended to be supported as creating a directory (especially since it's blocked in the pre-receive hook if you do it through the repo), but the error is unhandled:

   CreateGist, open /tmp/3123603247/dir/path.rs: no such file or directory
  

• Attempting to edit a Gist through the web UI seems to consistently encounter an error ParseGistForm, form has no files. I created a Gist with the name "Basic", public, with the filename "Basic" and the contents "Basic", and when I edited the Gist and added an extra word to the "File content" of "Basic" and hit "Save", I received this error. It looks to me like the <textarea> and <input> for the file are missing name attributes and so they aren't being posted. (This testing was with Firefox, and, to be fair, I merged the forgejo branch in as my DB schema level was higher than this branch; so this could be an artifact of that)

• When creating a new Gist and having the button options "Add file" and "Add" at the bottom seem a little too-similar and confusing (subjective). I get the difference but only after thinking about it a bit. If the button on the right was "Save" (like it is on edit), or maybe "Create Gist", it would feel a little clearer to me which button I need to press to continue forward.

• There doesn't seem to be a mechanism to delete a file from the Gist edit UI. Not a big deal, but thought I would mention it in case it was an oversight; certainly could be added later if there's a real need.

• Gists aren't linked/listed on the user's dashboard along with other repositories. I'm not sure if this is where most people would feel their navigation workflow would go through, but it seemed to make sense to me. With them not being present there, I thought that I might have lost the links to my Gists until I found them on Explore > Gists... I don't think I feel that "Explore" is where I'm going for my own contents typically?

• I wonder whether changing a Gist from "Public" to "Hidden" might have a subtle security risk; the URL for the Gist has already been exposed on the public Explore page, and it somewhat violates the expectation of "Hidden" (only available to those who you share the URL with) because it was previously public. Would it be possible for a change Public -> Hidden to regenerate the Gist's UUID? That would make it actually hidden.

• The web UI doesn't contain any access to the history of the gist. At first I was thinking, that's a reasonable thing to leave out, especially in a first revision. But since history is accessible from the git repo, I wonder if there might be a mismatch in expectations leading to a security risk - - I make a gist, put an access toke in it, realize it was a mistake, edit it - - but because the UI doesn't show "vers. 2" or access to v1 I might think I've removed the mistake and not realize it is still around in the repo. (This is pretty obscure... Just a thought... Maybe not worth much more thought)

• Calling the Gist's identifier a "UUID" when it is a truncated UUID feels a little surprising, particularly in the API definitions where it is exposed to developers -- when I saw gistuuid as an API parameter in the Swagger, I expected it really required a UUID and was confused that the search API didn't provide one. My thought would be, either truncate it and call it an ID (or identifier, locator, etc.), or, use a full UUID.

• A 2^32 search space on the truncated ID, global for the entire instance, seems a little on the small side to me for keeping hidden gists secret -- (削除) at 66 requests per second (ref. codeberg rate limits Codeberg/Community#425 (comment)) I can enumerate the entire search space in 2 years and extract all hidden gists. (2**32 / 66 / 3600 / 24 / 365 -> 2.063 yrs) (削除ここまで) Pardon me, made a calculation error in here as the rate limit is 1/10th of what I calculated here 😑, making it not as bad, but the security of the hidden gists would arguably be reliant on rate limiting.

Found one other thing that is a little more critical than some of those other points: CreateGist is vulnerable to a directory traversal attack which allows writes to arbitrary files. I put the file name ../../../../../../../../home/mfenniak/Dev/forgejo/test.txt in, and it exited the /tmp directory and performed the os.WriteFile to my arbitrary file. The addition to the git repo failed with an error, but the file was populated:

$ pwd
/home/mfenniak/Dev/forgejo
$ cat test.txt
abc

Later error, but the arbitrary file write is already done:

CreateGist, exit status 128 - fatal: ../../../../../../../../home/mfenniak/Dev/forgejo/test.txt: '../../../../../../../../home/mfenniak/Dev/forgejo/test.txt' is outside repository at '/tmp/1660668797'

Thanks @mfenniak for the detailed feedback.

I assume this isn't intended to be supported as creating a directory

I forgot to check it there. I will add it.

Attempting to edit a Gist through the web UI seems to consistently encounter an error

Looks like I accidentally broke this somehow. I will fix it.

or maybe "Create Gist"

I will rename the Button.

There doesn't seem to be a mechanism to delete a file from the Gist edit UI.

I will add that.

I'm not sure if this is where most people would feel their navigation workflow would go through, but it seemed to make sense to me

They are currently linked on the Profile and the Explore Page. I'm not sure, if I should add them to the Dashboard in this PR.

I wonder whether changing a Gist from "Public" to "Hidden" might have a subtle security risk;

Hidden is not meant to be perfectly save, It is more meant for things that are not interested for the public e.g. a Log that you want to share. There is no guarantee that it will not be indexed by a search engine.

But since history is accessible from the git repo, I wonder if there might be a mismatch in expectations leading to a security risk

History will not be part of this PR. I'm not sure if this should be added at all, as I never felt the need to browse the History of a Gist.

Calling the Gist's identifier a "UUID" when it is a truncated UUID feels a little surprising

At first it was a normal UUID, but I felt like this is to long for a URL, so I truncated it. Not sure, how I should call it instead, as ID is already a existing field.

CreateGist is vulnerable to a directory traversal attack which allows writes to arbitrary files.

This will be fixed, when add a check to ensure there is no directory in the name.

I think this needs to be ENABLED=false by default, at lease for the first time it appears in a major release.

I'm not so sure about that. If people want to test it, they can just use the Code from this PR.

I have now pushed the first Update to this PR:

• You can now edit Gists in the Web UI
• It is no longer possible to use a filename with a path separator
• Some Bugs were fixed
• A Sitemap has been added
• Tests were added wich already cover a lot of code. More will follow.

The latest Commit adds the following:

• Gists are now deleted when their Owner is deleted
• Gists can now be manged from the Admin Interface

Maybe dont use the Gist branding for the Gists tab, looks out of place tbh

Maybe dont use the Gist branding for the Gists tab,

The problem is that I don't have any other Icon

I couldn't find any evidence of Gist(s) being a GitHub trademark, not that I looked a lot, but it must be in the icon pack for a good reason, just like the copilot one.

This is one more reason I would suggest to rename the entire thing to Snippets.

I think you will be fine using https://primer.style/foundations/icons/file-code-16 or https://primer.style/foundations/icons/code-square-16 for an icon.

This is one more reason I would suggest to rename the entire thing to Snippets.

I also think that a more general, freer name is good. And Snippets is already known (from GitLab).
Alternatively, I would also be happy with Pastes (Sourcehut) - but I prefer Snippets.

I'd like to keep the name Gists for the following reasons:

• The name is git related. Gist is just git with a extra s.
• The name is already well known and also used outside GitHub (e.g. Opengist)
• I don't need to change a lot of code

Random user here—other than disagreement on the name, is there anything blocking this?

Marked as WIP: as it is not actively worked on at the moment.

This pull request contains good code but it has not seen any activity in a long time. In order to unclutter the queue and help reviewers focus on those that are actively worked on, it is now closed. Feel free to re-open it as soon as you want. It is also marked with the label valuable code so it can conveniently be found later and not confused with pull requests closed for other reasons.

@earl-warren I'm interested in continue this. What is exactly needed to merge this (at least in an experimental state)?

Since you're asking, working on the design of this new feature should be done before implementing it. There are a lot of moving parts and diving right into code may not be the best approach. Consider just this question: "What is the designed UX for a Gist feature"?

If you wonder what I mean by "design", this is an example.

working on the design of this new feature should be done before implementing it

It's already implemented since almost a year

I really appreciate you have put a lot of energy into it and it is kind of disappointing to contemplate the prospect of not having it done.

How do you know the UX is good? You imagined it and thought about it by yourself, which is not an easy task. I'm sure it will be a perfect match for you. But how do you know it will also be a good match for most of the Forgejo users? That is one of the difficult questions that working on design can clarify.

But how do you know it will also be a good match for most of the Forgejo users?

The UI is already there and working. Everyone can just test it and give feedback.

I'd be happy to apply and test it on my 13.0.2, but the patch as it stands does not seem to apply cleanly to that version (perhaps entirely unsurprisingly). What's the best way of going about testing it?

I fully understand and value the approach of designing a feature prior to implementing it. That said, perhaps in some cases the design proces could be permitted to be informed by working code and practical experimentation?

This is a feature I am greatly missing. The size of this contribution seems rather significant, and it seems a shame that it's not accepted as a WIP feature, at the very least.

@ltning I have updated the branch, so you can test it.

I know you refused rename the feature a year ago but in case you are up for reconsiderations:

• it is not well translatable, you can pretty much expect translators to either not translate it at all, leaving a meaningless name as is for their language, or that they will translate another word like "snippet", or "paste" or whatever else is popular
• in explore tabs, the gist icon is literally next to the word "gist". Neither icon nor the text give any idea to what the tab is about. The tab just says "Gist Gists". There are much more meaningful octicons available like file-code and log
• even if the the name Gist is referencing Git, it's not easily perceivable. It also doesn't need to be related to Git as the primary feature is quick saving of text snippets with less complexity than repositories have, not versioning