[BUG] Deployment via Docker and NFS fails #1590

2023年10月09日 23:31:01 +02:00

label

dbischof90 commented

2023年10月09日 23:34:55 +02:00

Author

Copy link

I want to add a little progress that I have made too:
If I change the UID and the GUID in the Dockerfile to match the anonuid/guid in the NFS export, I manage to resolve the /data folder rights issues:

daniel@server2:~ $ docker compose -f Software/forgejo.yaml up
[+] Running 1/1
 ✔ Container forgejo Recreated 0.2s
Attaching to forgejo
forgejo | Generating /data/ssh/ssh_host_ed25519_key...
forgejo | Generating /data/ssh/ssh_host_rsa_key...
forgejo | Generating /data/ssh/ssh_host_ecdsa_key...
forgejo | chown: /data/ssh/ssh_host_ecdsa_key: Operation not permitted
forgejo | chown: /data/ssh/ssh_host_ecdsa_key.pub: Operation not permitted
forgejo | chown: /data/ssh/ssh_host_ed25519_key: Operation not permitted
forgejo | chown: /data/ssh/ssh_host_ed25519_key.pub: Operation not permitted
forgejo | chown: /data/ssh/ssh_host_rsa_key: Operation not permitted
forgejo | chown: /data/ssh/ssh_host_rsa_key.pub: Operation not permitted
forgejo | Server listening on :: port 22.
[...]

However, in the Dockerfile, the git user is created with a different UID/GUID and can thus not chown the folder. However, since all users are mapped by the server to admin here, there is obviously no issue in actually creating all folders, however that still seems to be enough for the Docker file to just wipe everything (?)

Help 💯

I want to add a little progress that I have made too: If I change the UID and the GUID in the Dockerfile to match the `anonuid/guid` in the NFS export, I manage to resolve the `/data` folder rights issues: ```bash daniel@server2:~ $ docker compose -f Software/forgejo.yaml up [+] Running 1/1 ✔ Container forgejo Recreated 0.2s Attaching to forgejo forgejo | Generating /data/ssh/ssh_host_ed25519_key... forgejo | Generating /data/ssh/ssh_host_rsa_key... forgejo | Generating /data/ssh/ssh_host_ecdsa_key... forgejo | chown: /data/ssh/ssh_host_ecdsa_key: Operation not permitted forgejo | chown: /data/ssh/ssh_host_ecdsa_key.pub: Operation not permitted forgejo | chown: /data/ssh/ssh_host_ed25519_key: Operation not permitted forgejo | chown: /data/ssh/ssh_host_ed25519_key.pub: Operation not permitted forgejo | chown: /data/ssh/ssh_host_rsa_key: Operation not permitted forgejo | chown: /data/ssh/ssh_host_rsa_key.pub: Operation not permitted forgejo | Server listening on :: port 22. [...] ``` However, in the Dockerfile, the git user is created with a different UID/GUID and can thus not `chown` the folder. However, since all users are mapped by the server to admin here, there is obviously no issue in actually creating all folders, however that still seems to be enough for the Docker file to just wipe everything (?) Help 💯

earl-warren commented

2023年10月10日 00:02:33 +02:00

Contributor

Copy link

docker/root/etc/s6/gitea/setup chown if and only if the /data/gitea directory does not belong to the right user. Can you make it so it does?

[`docker/root/etc/s6/gitea/setup`](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/docker/root/etc/s6/gitea/setup#L63-L65) chown if and only if the `/data/gitea` directory does not belong to the right user. Can you make it so it does?

dbischof90 commented

2023年10月10日 07:20:05 +02:00

Author

Copy link

Wouldn’t it be enough to check whether the users can read and write in the folder instead of owning it? Which feature explicitly requires owning the file/folder structure?

Unfortunately not (uniformly). As discussed via chat, I can at best change the UID and the GUID to match the anonuid and in that case it works but only for the technical user, not the git user (see my last comment). Wouldn’t it be enough to check whether the users can read and write in the folder instead of owning it? Which feature explicitly requires owning the file/folder structure?

earl-warren commented

2023年10月10日 13:15:00 +02:00

Contributor

Copy link

for the technical user, not the git user

Can you make them both one and the same?

Wouldn’t it be enough to check whether the users can read and write in the folder instead of owning it?

That's not impossible in theory but the level of testing it would require to ensure it works consistently is significant. It is not unreasonable to ensure all files are owned by the same user, from the start. NFS can be configured to make it so the ownership is consistent, I'm sure.

I don't think Forgejo can be (or should be) modified to deal with files with unknown ownship or with ownership that is changed by something else (NFS or whatever). It is reasonable for a service to rely on a stable and predictable permission/ownership file tree.

> for the technical user, not the git user Can you make them both one and the same? > Wouldn’t it be enough to check whether the users can read and write in the folder instead of owning it? That's not impossible in theory but the level of testing it would require to ensure it works consistently is significant. It is not unreasonable to ensure all files are owned by the same user, from the start. NFS can be configured to make it so the ownership is consistent, I'm sure. I don't think Forgejo can be (or should be) modified to deal with files with unknown ownship or with ownership that is changed by something else (NFS or whatever). It is reasonable for a service to rely on a stable and predictable permission/ownership file tree.

dbischof90 commented

2023年10月10日 23:39:36 +02:00

Author

Copy link

In theory that could work. However

I need to create a user group with id 1000 on the server with sufficient rights
I need to create a user on the server with id 1000 with sufficient rights
Everytime I'm changing something in the GUI I need to manually change my export configuration and reexport the whole share again as the automatic anon ids are different and I don't think that this is a good solution.

Today I tried out other remote file systems:
CIFS:

has a user mapping but can also not chown a remote folder

NFSv4:

Same problem as with NFSv3, since here the UID matching is exchanged for the actual user name needing to exist on the remote system. On my system, the creation of a user named git is disabled for security reasons in the GUI, it would also need to receive the respective rights in the folders to read/write. So no real improvement.

NFSv4+Kerberos:

Haven't get that going yet - and it also feels a bit overengineering the problem.

SSHFS:

That almost worked, however this fusemount only has the nuclear option to allow_others if I want to change ownership, essentially granting every folder and file created automatically chmod +777. Besides this being a clear security problem, even the key generation stepped in rightfully:

[...]
forgejo | Unable to load host key "/data/ssh/ssh_host_ecdsa_key": bad permissions
forgejo | Unable to load host key: /data/ssh/ssh_host_ecdsa_key
forgejo | sshd: no hostkeys available -- exiting.
forgejo | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
forgejo | @ WARNING: UNPROTECTED PRIVATE KEY FILE! @
forgejo | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
forgejo | Permissions 0777 for '/data/ssh/ssh_host_ed25519_key' are too open.
forgejo | It is required that your private key files are NOT accessible by others.
forgejo | This private key will be ignored.
[...]

and the startup process did not proceed from there.

I understand your point. I think the opposite - Forgejo should not really mind whether or not it actually owns the file as it might (like in this case) not own the file system. I read now quite a bit about the rationale and that makes quite some sense if I think about it. I still think that being able to read and write files is enough for the full functionality. I also agree that a stable and predictable permission/ownership file tree is important but that doesn't imply it has to be the user itself.

EDIT:
What has to be considered in this scenario is „what does chown mean on a remote file system?" How is the file server supposed to interpret the ownership of the file when the user is not defined in their own /etc/passwd? That would leave the file in an invalid state hence it can’t be owned by anyone outside of the file server. To summarise this as a fixed rule: It must be owned by a user which is defined on the machine where the folder is physically located. That assumption is trivially fulfilled if all is on the same device but I’m now 95% sure that if we want to support data folders at remote locations via any systems then the docker file can’t rely on chown but must check whether the file server allows writing and reading.
EDIT END

I think I exhausted about 80-90% of the options out there and I currently don't know how to proceed - if what you say is true and it's a huge technical undertaking then that means no network drives can be involved here sadly.

In theory that could work. However - I need to create a user group with id 1000 on the server with sufficient rights - I need to create a user on the server with id 1000 with sufficient rights - Everytime I'm changing something in the GUI I need to manually change my export configuration and reexport the whole share again as the automatic anon ids are different and I don't think that this is a good solution. Today I tried out other remote file systems: CIFS: - has a user mapping but can also not `chown` a remote folder NFSv4: - Same problem as with NFSv3, since here the UID matching is exchanged for the actual user name needing to exist on the remote system. On my system, the creation of a user named `git` is disabled for security reasons in the GUI, it would also need to receive the respective rights in the folders to read/write. So no real improvement. NFSv4+Kerberos: - Haven't get that going yet - and it also feels a bit overengineering the problem. SSHFS: - That almost worked, however this `fusemount` only has the nuclear option to `allow_others` if I want to change ownership, essentially granting every folder and file created automatically `chmod +777`. Besides this being a clear security problem, even the key generation stepped in rightfully: ```bash [...] forgejo | Unable to load host key "/data/ssh/ssh_host_ecdsa_key": bad permissions forgejo | Unable to load host key: /data/ssh/ssh_host_ecdsa_key forgejo | sshd: no hostkeys available -- exiting. forgejo | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ forgejo | @ WARNING: UNPROTECTED PRIVATE KEY FILE! @ forgejo | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ forgejo | Permissions 0777 for '/data/ssh/ssh_host_ed25519_key' are too open. forgejo | It is required that your private key files are NOT accessible by others. forgejo | This private key will be ignored. [...] ``` and the startup process did not proceed from there. I understand your point. I think the opposite - Forgejo should not really mind whether or not it actually owns the file as it might (like in this case) not own the file system. I read now quite a bit about the rationale and that makes quite some sense if I think about it. I still think that being able to read and write files is enough for the full functionality. I also agree that a stable and predictable permission/ownership file tree is important but that doesn't imply it has to be the user itself. *EDIT:* What has to be considered in this scenario is „what does `chown` mean on a remote file system?" How is the file server supposed to interpret the ownership of the file when the user is not defined in their own `/etc/passwd`? That would leave the file in an invalid state hence it can’t be owned by anyone outside of the file server. To summarise this as a fixed rule: **It *must* be owned by a user which is defined on the machine where the folder is physically located**. That assumption is trivially fulfilled if all is on the same device but I’m now 95% sure that if we want to support data folders at remote locations via any systems then the docker file can’t rely on `chown` but must check whether the file server allows writing and reading. *EDIT END* I think I exhausted about 80-90% of the options out there and I currently don't know how to proceed - if what you say is true and it's a huge technical undertaking then that means no network drives can be involved here sadly.

earl-warren commented

2023年10月11日 12:33:13 +02:00

Contributor

Copy link

Forgejo should not really mind whether or not it actually owns the file...

In theory, I agree it should be fine. But there are corner cases where mixed ownership of directories and files become a security concern. The one example relevant to Forgejo is from git https://github.blog/2022-04-12-git-security-vulnerability-announced/.

I’m now 95% sure that if we want to support data folders at remote locations via any systems then the docker file can’t rely on chown but must check whether the file server allows writing and reading.

Note that chown is only run if the ownership of the files is verified to be inconsistent. If the remote file system is configured sanely to ensure that, chown won't get involved.

Everytime I'm changing something in the GUI I need to manually change my export configuration and reexport the whole share again as the automatic anon ids are different and I don't think that this is a good solution.

So, if I understand correctly, you figured out a way to ensure that all files are owned by a single user and solve that problem in a durable way. But what you did is undone when you use the GUI, which is a different problem.

> Forgejo should not really mind whether or not it actually owns the file... In theory, I agree it should be fine. But there are corner cases where mixed ownership of directories and files become a security concern. The one example relevant to Forgejo is from git https://github.blog/2022-04-12-git-security-vulnerability-announced/. > I’m now 95% sure that if we want to support data folders at remote locations via any systems then the docker file can’t rely on chown but must check whether the file server allows writing and reading. Note that chown is only run if the ownership of the files is verified to be inconsistent. If the remote file system is configured sanely to ensure that, chown won't get involved. > Everytime I'm changing something in the GUI I need to manually change my export configuration and reexport the whole share again as the automatic anon ids are different and I don't think that this is a good solution. So, if I understand correctly, you figured out a way to ensure that all files are owned by a single user and solve that problem in a durable way. But what you did is undone when you use the GUI, which is a different problem.

dbischof90 commented

2023年10月11日 13:42:13 +02:00

Author

Copy link

In theory, I agree it should be fine. But there are corner cases where mixed ownership of directories and files become a security concern. The one example relevant to Forgejo is from git https://github.blog/2022-04-12-git-security-vulnerability-announced/.

Hm, that makes it indeed tricky. However, that specific item seems to be fixed?

I’m now 95% sure that if we want to support data folders at remote locations via any systems then the docker file can’t rely on chown but must check whether the file server allows writing and reading.

Note that chown is only run if the ownership of the files is verified to be inconsistent. If the remote file system is configured sanely to ensure that, chown won't get involved.

Can you explain what 'inconsistent' means here? I can imagine a few ways here but, again, maybe it's not relevant at all who owns it as long as you can write in it. Excluding security cases as the one above - but those cases would then affect all systems where any network share is involved, that would be the majority of computer systems. We do have processes with non-local file systems, they must do this somehow.

Everytime I'm changing something in the GUI I need to manually change my export configuration and reexport the whole share again as the automatic anon ids are different and I don't think that this is a good solution.

So, if I understand correctly, you figured out a way to ensure that all files are owned by a single user and solve that problem in a durable way. But what you did is undone when you use the GUI, which is a different problem.

That is almost correct - I didn't test it but if I would, then that might actually work. The issue that stopped me was the git user again - since I have two users in the docker file I can't consistently map them to one UID without diving into the docker file and the source code and seeing where and whether that user does some specific shenanigans which the other user doesn't do.

I currently see two things, one I can try and one you could answer or investigate:

One way to map UID/GUIDs between systems is with an authentification such as Kerberos which I didn't pursue so far out of personal lazyness and zero experience with Kerberos. I might be able to map a remote user 'repo' which can write in my remote folder and map forgejo_user:repo and git:repo. I also don't want to give up on CIFS just yet, I will try out later if something along the line of mount -t cifs -o username=****,password="****",uid=repo works
It might make the above scenario resolvable if it's enough that the user groups coincide and chgrp would suffice? Since I can provide an anonguid that I could map both the git user and the forgejo user into. If then the git user was added to the forgejo user group instead of the group 1000, that might resolve our issue...

> In theory, I agree it should be fine. But there are corner cases where mixed ownership of directories and files become a security concern. The one example relevant to Forgejo is from git https://github.blog/2022-04-12-git-security-vulnerability-announced/. Hm, that makes it indeed tricky. However, that specific item seems to be fixed? > > I’m now 95% sure that if we want to support data folders at remote locations via any systems then the docker file can’t rely on chown but must check whether the file server allows writing and reading. > > Note that chown is only run if the ownership of the files is verified to be inconsistent. If the remote file system is configured sanely to ensure that, chown won't get involved. Can you explain what 'inconsistent' means here? I can imagine a few ways here but, again, maybe it's not relevant at all who owns it as long as you can write in it. Excluding security cases as the one above - but those cases would then affect all systems where any network share is involved, that would be the majority of computer systems. We do have processes with non-local file systems, they must do this somehow. > > Everytime I'm changing something in the GUI I need to manually change my export configuration and reexport the whole share again as the automatic anon ids are different and I don't think that this is a good solution. > > So, if I understand correctly, you figured out a way to ensure that all files are owned by a single user and solve that problem in a durable way. But what you did is undone when you use the GUI, which is a different problem. That is almost correct - I didn't test it but if I would, then that might actually work. The issue that stopped me was the git user again - since I have two users in the docker file I can't consistently map them to one UID without diving into the docker file and the source code and seeing where and whether that user does some specific shenanigans which the other user doesn't do. I currently see two things, one I can try and one you could answer or investigate: 1) One way to map UID/GUIDs between systems is with an authentification such as Kerberos which I didn't pursue so far out of personal lazyness and zero experience with Kerberos. I might be able to map a remote user 'repo' which can write in my remote folder and map forgejo_user:repo and git:repo. I also don't want to give up on CIFS just yet, I will try out later if something along the line of `mount -t cifs -o username=****,password="****",uid=repo` works 2) It might make the above scenario resolvable if it's enough that the user **groups** coincide and `chgrp` would suffice? Since I can provide an anonguid that I could map both the git user and the forgejo user into. If then the git user was added to the forgejo user group instead of the group 1000, that might resolve our issue...

earl-warren commented

2023年10月11日 14:03:12 +02:00

Contributor

Copy link

In theory, I agree it should be fine. But there are corner cases where mixed ownership of directories and files become a security concern. The one example relevant to Forgejo is from git https://github.blog/2022-04-12-git-security-vulnerability-announced/.

Hm, that makes it indeed tricky. However, that specific item seems to be fixed?

Yes. And the fix involves git throwing an error if the ownership of some directories is not as expected. My point here is less about that specific case and more about the notion that ownership of files and directories matter in some cases. it is not exclusively about being able to read and write the files.

Can you explain what 'inconsistent' means here?

Some directories not being owned by the user running Forgejo.

Everytime I'm changing something in the GUI I need to manually change my export configuration and reexport the whole share again as the automatic anon ids are different and I don't think that this is a good solution.

So, if I understand correctly, you figured out a way to ensure that all files are owned by a single user and solve that problem in a durable way. But what you did is undone when you use the GUI, which is a different problem.

That is almost correct - I didn't test it but if I would, then that might actually work. The issue that stopped me was the git user again - since I have two users in the docker file I can't consistently map them to one UID without diving into the docker file and the source code and seeing where and whether that user does some specific shenanigans which the other user doesn't do.

There is only one user in the docker image, why do you have two? What are they.

It might make the above scenario resolvable if it's enough that the user groups coincide and chgrp would suffice? Since I can provide an anonguid that I could map both the git user and the forgejo user into. If then the git user was added to the forgejo user group instead of the group 1000, that might resolve our issue...

I do not think it is realistic to expect a change in how permissions are set in Forgejo. It is currently very simple and secure: all files and directories have to be owned by the user running Forgejo.

But since you figured out a way to make that happen with the remote file system (as long as you do not use the GUI which, for some reason, will mess things up), that should not be a blocker.

> > In theory, I agree it should be fine. But there are corner cases where mixed ownership of directories and files become a security concern. The one example relevant to Forgejo is from git https://github.blog/2022-04-12-git-security-vulnerability-announced/. > > Hm, that makes it indeed tricky. However, that specific item seems to be fixed? Yes. And the fix involves git throwing an error if the ownership of some directories is not as expected. My point here is less about that specific case and more about the notion that ownership of files and directories matter in some cases. it is not exclusively about being able to read and write the files. > Can you explain what 'inconsistent' means here? Some directories not being owned by the user running Forgejo. > > > Everytime I'm changing something in the GUI I need to manually change my export configuration and reexport the whole share again as the automatic anon ids are different and I don't think that this is a good solution. > > > > So, if I understand correctly, you figured out a way to ensure that all files are owned by a single user and solve that problem in a durable way. But what you did is undone when you use the GUI, which is a different problem. > > That is almost correct - I didn't test it but if I would, then that might actually work. The issue that stopped me was the git user again - since I have two users in the docker file I can't consistently map them to one UID without diving into the docker file and the source code and seeing where and whether that user does some specific shenanigans which the other user doesn't do. There is only one user in the docker image, why do you have two? What are they. > 2) It might make the above scenario resolvable if it's enough that the user **groups** coincide and `chgrp` would suffice? Since I can provide an anonguid that I could map both the git user and the forgejo user into. If then the git user was added to the forgejo user group instead of the group 1000, that might resolve our issue... I do not think it is realistic to expect a change in how permissions are set in Forgejo. It is currently very simple and secure: all files and directories have to be owned by the user running Forgejo. But since you figured out a way to make that happen with the remote file system (as long as you do not use the GUI which, for some reason, will mess things up), that should not be a blocker.

dbischof90 commented

2023年10月11日 14:41:17 +02:00

Author

Copy link

I do not think it is realistic to expect a change in how permissions are set in Forgejo. It is currently very simple and secure: all files and directories have to be owned by the user running Forgejo.

Okay, in that case also mapping won't change a thing as even with Kerberos I just map user IDs back and forwards. That means there is no way to run Gitea or Forgejo with data on remote systems.

But since you figured out a way to make that happen with the remote file system (as long as you do not use the GUI which, for some reason, will mess things up), that should not be a blocker.

Hmmm... in theory. As I mentioned, I took my initial experiment as an indicator of whether that resolves the problem, which it did. The problem is the second user...

There is only one user in the docker image, why do you have two? What are they.

... which is created here (the git user):

Dockerfile

Line 62 in 250bf38

adduser \

...

... wait.

Quick reversal back, I was about to write that the second user fails since it can't be mapped to a remote user with the same ID as referenced here (#1590 (comment)) but it is actually in fact the ssh folder that makes problems, not the git folder which I'm just realising. Which also means that I don't grasp the full problem yet so there is still some hope - plus your statement

It is currently very simple and secure: all files and directories have to be owned by the user running Forgejo.

does in fact currently also not apply since the file is actually not owned by the forgejo user on the physical mount as no such user exists there, it's just that NFS maps all user data to an anonymous user which has the same user id as forgejo and maps that user to the root user on the file system. So we already are at a point where NFS just pretends Forgejo would own the data, which works in fact. That's good, that means it could maybe work!

Can we try to identify where that SSH folder comes from and what makes it special? And would it hurt if that folder was also owned by the Forgejo user?

> I do not think it is realistic to expect a change in how permissions are set in Forgejo. It is currently very simple and secure: all files and directories have to be owned by the user running Forgejo. Okay, in that case also mapping won't change a thing as even with Kerberos I just map user IDs back and forwards. That means there is no way to run Gitea or Forgejo with data on remote systems. > But since you figured out a way to make that happen with the remote file system (as long as you do not use the GUI which, for some reason, will mess things up), that should not be a blocker. Hmmm... in theory. As I mentioned, I took my initial experiment as an indicator of whether that resolves the problem, which it did. The problem is the second user... > There is only one user in the docker image, why do you have two? What are they. ... which is created here (the `git` user): https://codeberg.org/forgejo/forgejo/src/commit/250bf3890f0a3a8b209f10d59cb26bca2ee2f719/Dockerfile#L62 ... ... wait. Quick reversal back, I was about to write that the second user fails since it can't be mapped to a remote user with the same ID as referenced here (https://codeberg.org/forgejo/forgejo/issues/1590#issuecomment-1270166) but it is actually in fact the `ssh` folder that makes problems, not the `git` folder which I'm just realising. Which also means that I don't grasp the full problem yet so there is still some hope - plus your statement > It is currently very simple and secure: all files and directories have to be owned by the user running Forgejo. does in fact currently also not apply since the file is actually not owned by the forgejo user on the physical mount as no such user exists there, it's just that NFS maps all user data to an anonymous user which has the same user id as forgejo and maps that user to the root user on the file system. So we already are at a point where NFS just pretends Forgejo would own the data, which works in fact. That's good, that means it could maybe work! Can we try to identify where that SSH folder comes from and what makes it special? And would it hurt if that folder was also owned by the Forgejo user?

earl-warren commented

2023年10月11日 15:09:53 +02:00

Contributor

Copy link

There is only one user in the docker image, why do you have two? What are they.

... which is created here (the git user):
Dockerfile
Line 62 in 250bf38
adduser \

That is the only user.

Can we try to identify where that SSH folder comes from and what makes it special? And would it hurt if that folder was also owned by the Forgejo user?

Can you let me know what are the two users. There is one, which is git as you figured out. What is the other one?

> > There is only one user in the docker image, why do you have two? What are they. > > ... which is created here (the `git` user): https://codeberg.org/forgejo/forgejo/src/commit/250bf3890f0a3a8b209f10d59cb26bca2ee2f719/Dockerfile#L62 > That is the only user. > Can we try to identify where that SSH folder comes from and what makes it special? And would it hurt if that folder was also owned by the Forgejo user? Can you let me know what are the **two** users. There is one, which is `git` as you figured out. What is the other one?

dbischof90 commented

2023年10月11日 16:38:14 +02:00

Author

Copy link

Hm. Indeed - apologies. I thought there was a second user but in fact (https://codeberg.org/forgejo/forgejo/src/branch/forgejo/docker/root/usr/bin/entrypoint) the UID/GUIDs are just changed lateron.

IDK, this all apparently almost works, even without an advanced user mapping via network, the only exception seems to be the SSH setup (which happens if I see that correctly here: https://codeberg.org/forgejo/forgejo/src/branch/forgejo/docker/root/etc/s6/openssh/setup). What happens if the key generation is happening outside of /data in a temporary folder but then later moved there and owned by $USER_ID? Would that affect the functionality? If that's feasible, that might resolve it...

Hm. Indeed - apologies. I thought there was a second user but in fact (https://codeberg.org/forgejo/forgejo/src/branch/forgejo/docker/root/usr/bin/entrypoint) the UID/GUIDs are just changed lateron. IDK, this all apparently almost works, even without an advanced user mapping via network, the only exception seems to be the SSH setup (which happens if I see that correctly here: https://codeberg.org/forgejo/forgejo/src/branch/forgejo/docker/root/etc/s6/openssh/setup). What happens if the key generation is happening outside of `/data` in a temporary folder but then later moved there and owned by `$USER_ID`? Would that affect the functionality? If that's feasible, that might resolve it...

earl-warren commented

2023年10月11日 17:24:17 +02:00

Contributor

Copy link

If that's impossible for some reasons there are two solutions:

you store /data/ssh on a local file system so it is owned by root (the content is tiny and won't change over the lifetime of the instance)
you switch to using a rootless container that runs with an embedded ssh server by default (START_SSH_SERVER)

I'm glad this part is sorted out. Now for the ssh server, is there any way you can convince NFS that these files are owned by root? If yes, there will still be errors when it tries to chown them on every start but this is harmless (and could be fixed in a PR that verifies the desired permissions are what they should be before attempting to chown). If that's impossible for some reasons there are two solutions: * you store `/data/ssh` on a local file system so it is owned by root (the content is tiny and won't change over the lifetime of the instance) * you switch to using a rootless container that runs with an embedded ssh server by default ([START_SSH_SERVER](https://forgejo.org/docs/latest/admin/config-cheat-sheet/#server-server))

👍 1

dbischof90 commented

2023年10月11日 17:34:03 +02:00

Author

Copy link

Those are actually things I will try out! And report back here.
If the former works, would that be something you'd let to be configurable upstream?
Otherwise, true, there's still the rootless container, that can be an option too that I haven't considered yet. Are there any implications here?

Those are actually things I will try out! And report back here. If the former works, would that be something you'd let to be configurable upstream? Otherwise, true, there's still the rootless container, that can be an option too that I haven't considered yet. Are there any implications here?

👍 1

earl-warren commented

2023年10月11日 17:46:09 +02:00

Contributor

Copy link

Otherwise, true, there's still the rootless container, that can be an option too that I haven't considered yet. Are there any implications here?

Other than the ssh server it is the same binary.

> Otherwise, true, there's still the rootless container, that can be an option too that I haven't considered yet. Are there any implications here? Other than the ssh server it is the same binary.

dbischof90 commented

2023年10月11日 22:21:39 +02:00

Author

Copy link

Otherwise, true, there's still the rootless container, that can be an option too that I haven't considered yet. Are there any implications here?

Other than the ssh server it is the same binary.

Success! This worked. The rootless container (drop-in replacement in the compose file, even if it's not made with that purpose in mind) and matching ANONUID/GUID with USER_ID/USER_GID worked, I have now a persistent state via NFS. Thank you!

> > Otherwise, true, there's still the rootless container, that can be an option too that I haven't considered yet. Are there any implications here? > > Other than the ssh server it is the same binary. Success! This worked. The rootless container (drop-in replacement in the compose file, even if it's not made with that purpose in mind) and matching ANONUID/GUID with USER_ID/USER_GID worked, I have now a persistent state via NFS. Thank you!

❤️ 1

earl-warren commented

2023年10月11日 22:40:06 +02:00

Contributor

Copy link

You've been exceptionally patient. This issue will be useful to other admins facing permission problems with remote file systems. Before today I've read it surface from time to time but without a definitive solution I could bookmark.

Shall we close this?

earl-warren commented

2023年10月11日 22:41:27 +02:00

Contributor

Copy link

If you have some fuel left, it would deserve an addition to the admin documentation, articulating the problem and the solution as well as the rationale for it while it is fresh in your mind 🙏

earl-warren commented

2023年10月11日 22:41:45 +02:00

Contributor

Copy link

https://codeberg.org/forgejo/docs/src/branch/next/docs/admin is where it can be contributed.

dbischof90 commented

2023年10月11日 22:57:53 +02:00

Author

Copy link

No worries, least I can do. You were very helpful :) I'll contribute and we can close this issue with a link to the post later so that others can read up.

❤️ 1

dbischof90 referenced this issue from forgejo/docs

2023年10月14日 15:31:11 +02:00

Documenting remote data folder #193

dbischof90 commented

2023年10月14日 15:33:26 +02:00

Author

Copy link

PR is open: forgejo/docs#193

PR is open: https://codeberg.org/forgejo/docs/pulls/193

dbischof90 closed this issue

2023年10月14日 15:33:29 +02:00

earl-warren referenced this issue from a commit

2023年10月14日 19:42:42 +02:00

admin: installation: remote data folder (#193)

earl-warren referenced this issue from a commit

2023年10月14日 19:46:26 +02:00

admin: installation: remote data folder (#193)

earl-warren referenced this issue from forgejo/docs

2023年10月14日 19:48:16 +02:00

admin: installation: remote data folder (#193) #194

fnetX referenced this issue

2026年01月06日 18:01:48 +01:00

problem: NFS "Operation not permitted" #10718

No Branch/Tag specified

Branches Tags

forgejo

renovate/forgejo-citation-js-monorepo

v16.0/forgejo

renovate/forgejo-github.com-urfave-cli-v3-3.x

renovate/forgejo-github.com-alecthomas-chroma-v2-2.x

renovate/forgejo-sharp-0.x

renovate/forgejo-github.com-sourcegraph-zoekt-digest

v15.0/forgejo

bp-v16.0/forgejo-74f6dda

v11.0/forgejo

renovate/forgejo-github.com-go-swagger-go-swagger-cmd-swagger-0.x

bp-v15.0/forgejo-daaee9c

bp-v15.0/forgejo-5c61808

bp-v15.0/forgejo-6cd3f02

v14.0/forgejo

bp-v15.0/forgejo-2176403

bp-v14.0/forgejo-9767ceb

bp-v14.0/forgejo-d996dfb

v13.0/forgejo

bp-v13.0/forgejo-dc0a63e

bp-v13.0/forgejo-e7ef2eb

v12.0/forgejo

bp-v12.0/forgejo-cf1fda8-a511e37

bp-v11.0/forgejo-b52cec7

v7.0/forgejo

v10.0/forgejo

v9.0/forgejo

v8.0/forgejo

v1.21/forgejo

v1.20/forgejo

v1.19/forgejo

v1.18/forgejo

v1.17/forgejo

v15.0.4

v11.0.16

v17.0.0-dev

v15.0.3

v11.0.15

v15.0.2

v11.0.14

v15.0.1

v14.0.5

v11.0.13

v15.0.0

v14.0.4

v11.0.12

v16.0.0-dev

v14.0.3

v11.0.11

v14.0.2

v14.0.1

v13.0.5

v11.0.10

v14.0.0

v13.0.4

v11.0.9

v15.0.0-dev

v13.0.3

v11.0.8

v13.0.2

v11.0.7

v13.0.1

v13.0.0

v14.0.0-dev

v12.0.4

v11.0.6

v12.0.3

v11.0.5

v12.0.2

v11.0.4

v12.0.1

v12.0.0

v11.0.3

v7.0.16

v13.0.0-dev

v11.0.2

v11.0.1

v7.0.15

v11.0.0

v12.0.0-dev

v10.0.3

v10.0.2

v7.0.14

v10.0.1

v7.0.13

v10.0.0

v11.0.0-dev

v9.0.3

v7.0.12

v9.0.2

v7.0.11

v9.0.1

v7.0.10

v9.0.0

v10.0.0-dev

v8.0.3

v7.0.9

v8.0.2

v7.0.8

v8.0.1

v7.0.7

v8.0.0

v7.0.6

v7.0.5

v9.0.0-dev

v7.0.4

v1.21.11-2

v7.0.3

v7.0.2

v7.0.1

v7.0.0

v1.21.11-1

v1.21.11-0

v1.21.10-0

v8.0.0-dev

v1.21.8-0

v1.21.7-0

v1.21.6-0

v1.21.5-0

v1.21.4-0

v1.21.3-0

v1.20.6-1

v1.21.2-1

v1.21.2-0

v1.20.6-0

v1.21.1-0

v1.20.5-1

v1.21.0-rc2

v1.21.0-rc1

v1.20.5-0

v7.0.0-dev

v1.21.0-rc0

v1.20.4-1

v1.20.4-0

v1.20.3-0

v1.20.2-0

v1.20.1-0

v1.20.0

v1.19.4-0

v1.21.0-dev

v1.20.0-rc2

v1.19.3-0

v1.19.2-0

v1.19.1-0

v1.19.0-3

v1.19.0-2

v1.18.5-0

v1.20.0-dev

v1.19.0-rc0

v1.18.3-2

v1.18.3-1

v1.18.3-0

v1.18.2-1

v1.18.2-0

v1.18.1-0

v1.18.0-1

v1.18.0-0

v1.18.0-rc1-2

v1.18.0-rc1-1

v1.18.0-rc1

v1.18.0-rc0

v1.19.0-dev

v1.17.3

v1.17.2

v1.17.1

v1.17.0

v1.17.0-rc2

v1.16.9

v1.17.0-rc1

v1.18.0-dev

v1.16.8

v1.16.7

v1.16.6

v1.16.5

v1.16.4

v1.16.3

v1.16.2

v1.16.1

v1.16.0

v1.15.11

v1.17.0-dev

v1.16.0-rc1

v1.15.10

v1.15.9

v1.15.8

v1.15.7

v1.15.6

v1.15.5

v1.15.4

v1.15.3

v1.15.2

v1.15.1

v1.14.7

v1.15.0

v1.15.0-rc3

v1.14.6

v1.15.0-rc2

v1.14.5

v1.16.0-dev

v1.15.0-rc1

v1.14.4

v1.14.3

v1.14.2

v1.14.1

v1.14.0

v1.13.7

v1.14.0-rc2

v1.13.6

v1.13.5

v1.14.0-rc1

v1.15.0-dev

v1.13.4

v1.13.3

v1.13.2

v1.13.1

v1.13.0

v1.12.6

v1.13.0-rc2

v1.14.0-dev

v1.13.0-rc1

v1.12.5

v1.12.4

v1.12.3

v1.12.2

v1.12.1

v1.11.8

v1.12.0

v1.11.7

v1.12.0-rc2

v1.11.6

v1.12.0-rc1

v1.13.0-dev

v1.11.5

v1.11.4

v1.11.3

v1.10.6

v1.12.0-dev

v1.11.2

v1.10.5

v1.11.1

v1.10.4

v1.11.0

v1.11.0-rc2

v1.10.3

v1.11.0-rc1

v1.10.2

v1.10.1

v1.10.0

v1.9.6

v1.9.5

v1.10.0-rc2

v1.11.0-dev

v1.10.0-rc1

v1.9.4

v1.9.3

v1.9.2

v1.9.1

v1.9.0

v1.9.0-rc2

v1.10.0-dev

v1.9.0-rc1

v1.8.3

v1.8.2

v1.8.1

v1.8.0

v1.8.0-rc3

v1.7.6

v1.8.0-rc2

v1.7.5

v1.8.0-rc1

v1.9.0-dev

v1.7.4

v1.7.3

v1.7.2

v1.7.1

v1.7.0

v1.7.0-rc3

v1.6.4

v1.7.0-rc2

v1.6.3

v1.7.0-rc1

v1.7.0-dev

v1.6.2

v1.6.1

v1.6.0

v1.6.0-rc2

v1.5.3

v1.6.0-rc1

v1.6.0-dev

v1.5.2

v1.5.1

v1.5.0

v1.5.0-rc2

v1.5.0-rc1

v1.5.0-dev

v1.4.3

v1.4.2

v1.4.1

v1.4.0

v1.4.0-rc3

v1.4.0-rc2

v1.3.3

v1.4.0-rc1

v1.3.2

v1.3.1

v1.3.0

v1.3.0-rc2

v1.3.0-rc1

v1.2.3

v1.2.2

v1.2.1

v1.2.0

v1.2.0-rc3

v1.2.0-rc2

v1.1.4

v1.2.0-rc1

v1.1.3

v1.1.2

v1.1.1

v1.1.0

v1.0.2

v1.0.1

v1.0.0

v0.9.99

Labels

Clear labels

arch

riscv64

Archived

backport/v1.19

Scheduled for backport to Forgejo v1.19

Archived

backport/v1.20

Scheduled for backport to Forgejo v1.20

Archived

backport/v1.21/forgejo

Scheduled for backport to Forgejo v1.21

Archived

backport/v10.0/forgejo

Automated backport to v10.0

Archived

backport/v11.0/forgejo

Automated backport to v11.0

backport/v12.0/forgejo

Automated backport to v12.0

Archived

backport/v13.0/forgejo

Automated backport to v13.0

Archived

backport/v14.0/forgejo

Automated backport to v14.0

Archived

backport/v15.0/forgejo

Automated backport to v15.0

backport/v16.0/forgejo

Automated backport to v16.0

backport/v7.0/forgejo

Scheduled for backport to Forgejo v7.0

Archived

backport/v8.0/forgejo

Scheduled for backport to Forgejo v8.0

Archived

backport/v9.0/forgejo

Scheduled for backport to Forgejo v9.0

Archived

breaking

The release containing this change is not backward compatible

Something is not working

Archived

it can be reproduced

bug has already been reported in the Forgejo tracker

needs-more-info

the information provided does not contain enough details

new-report

bug has just been reported and need triage (default label on issue creation)

reported-upstream

bug cannot be fixed within Forgejo easily, it has been reported upstream

code/actions

Forgejo Actions feature

code/api

API

code/auth

Forgejo Authentication

code/auth/faidp

Forgejo as Identity Provider (in OAuth/OIDC flow)

code/auth/farp

Forgejo as Relying Party / Client (in OAuth/OIDC flow)

code/email

Everything related to email in Forgejo

code/federation

Federation

code/git

Related to the Git backend in Forgejo

code/migrations

Migration between Git forges (i.e. for GitHub, GitLab, Gitea, Forgejo, etc.). NOT for database migrations.

code/packages

Forgejo package and container registry

code/wiki

database

MySQL

https://www.mysql.com/

database

PostgreSQL

https://www.postgresql.org/

database

SQLite

https://sqlite.org/

dependency-upgrade

https://codeberg.org/forgejo/forgejo/issues/2779

https://github.com/go-chi/chi/

Chi

Archived

https://github.com/alecthomas/chroma/

Chroma

Archived

https://f3.forgefriends.org/

https://forgefed.org

https://git.deuxfleurs.fr/Deuxfleurs/garage

https://github.com/go-gitea/gitea

Gitea

Archived

https://go.dev/

This issue or pull request already exists

enhancement/feature

New feature

forgejo/accessibility

Accessibility (a11y)

forgejo/branding

Branding (logo, name, tagline etc.)

forgejo/ci

Forgejo Actions CI configuration

forgejo/commit-graph

The commit graph feature and page.

forgejo/documentation

forgejo/furnace cleanup

Keeping Forgejo in sync with its dependencies and contributing back to them

Archived

forgejo/i18n

t9n/translation, l10n/localization, and i18n/internationalization of Forgejo

forgejo/interop

Interoperability with other services: Webhooks, bridges, integrations

Moderation

Privacy first

Release management

Performance and scaling

forgejo/security

Security (please disclose responsibly)

forgejo/ui

User interface

High

User research provides indicators that this would be good to have, interested contributors are encouraged to pick this.

Nice to have

This is likely worth having, but the assumption is not backed by user research data (it might benefit a small amount of users only.) Unlikely to receive much attention, but feel free to pick.

Undefined

Not enough information to assess the request's benefits. This issue may be closed if no gain is established: You can help by giving us more input.

Very High

User research indicates that this is an important improvement for Forgejo users. Contributions very welcome!

good first issue

Optimal for first-timers! Make sure to look for further explanations and ask for help if needed. If you want, you can consider the person who added this label as a point of contact.

i18n/backport-stable

This PR needs to be backported to stable branch of Forgejo safely and manually, using a migration script.

large

Large impact: Potential data loss, many users affected, major degradation in UX.

medium

Medium impact: Several users affected, degradation in UX, workarounds might be available but inconvenient.

small

Small impact: No data loss, workarounds might be available, affects few users.

Pull requests that have been merged with a manual test

unknown

Report was not yet triaged to assess impact.

Incompatible license

This pull request contains changes that are not (yet) compatible with the current Forgejo license

issue

closed

The issue was resolved in the repository of the dependency

issue

do-not-exist-yet

An issue should be created in the respository of the dependency

issue

open

An open issue exists in the upstream repository of the dependency

manual test

Archived

Manually tested during feature freeze

The manual test instructions were followed

FreeBSD

Specific to the FreeBSD Operating System

Linux

Specific to (GNU/)Linux Operating Systems

macOS

Specific to the MacOS Operating System

Windows

Specific to the Windows Operating System

problem

A user report about a problem. Needs to be triaged to find potential solutions.

regression

found in the version of the milestone and not before

release blocker

Issues that must be fixed before the release can be published

Release Cycle

Feature Freeze

Only bug fixes with automated tests (except for CSS/JavaScript)

Issues that must be fixed before Forgejo v7.0 can be released 17 April 2024

v7.0

Archived

Issues that must be fixed before Forgejo v7.0.1 can be released

v7.0.1

Archived

Issues that must be fixed before Forgejo v7.0.2 can be released

v7.0.2

Archived

Issues that must be fixed before Forgejo v7.0.3 can be released

v7.0.3

Archived

Issues that must be fixed before Forgejo v7.0.4 can be released

v7.0.4

Archived