Support for bind mounts #223

Alexis commented 2023年04月19日 02:22:54 +02:00 (Migrated from code.blicky.net)

I have to say upfront,
I LOVE YOU MAN!

The tool you made is ABSOLUTELY useful to us,
so much so that the first thing I though was "I've got to donate to this man!!!".

Leaving that aside,
I was traversing our CentOS-WHM system,
and I saw "/usr/share/cagefs-skeleton/var/lib/mysql - 241.2 GiB"
If you, as me, knew that "/usr/share/cagefs-skeleton" is a dir
that gets copied to all new accounts, I think you too would freak out :D
BUT, here is the catch:
The folder "/usr/share/cagefs-skeleton/var/lib/mysql" is a mount
(a mount in the SAME file system, in the same disk...)
But there is no FILE FLAG, like !, >, e, and no option to exclude mounts.

Cmd I run was:
ncdu -x --exclude-kernfs --no-follow-symlinks --group-directories-first --confirm-quit /

I have to say upfront, I LOVE YOU MAN! The tool you made is ABSOLUTELY useful to us, so much so that the first thing I though was "I've got to donate to this man!!!". Leaving that aside, I was traversing our CentOS-WHM system, and I saw "/usr/share/cagefs-skeleton/var/lib/mysql - 241.2 GiB" If you, as me, knew that "/usr/share/cagefs-skeleton" is a dir that gets copied to all new accounts, I think you too would freak out :D BUT, here is the catch: The folder "/usr/share/cagefs-skeleton/var/lib/mysql" is a mount (a mount in the SAME file system, in the same disk...) But there is no FILE FLAG, like !, >, e, and no option to exclude mounts. Cmd I run was: ncdu -x --exclude-kernfs --no-follow-symlinks --group-directories-first --confirm-quit /

yorhel commented 2023年04月19日 07:07:54 +02:00 (Migrated from code.blicky.net)

Yup, bind mounts are a pain and can't be detected with typical tooling. du -x and find -xdev etc will likely run into the same problem. I think this could be solved by parsing /proc/self/mounts and adding excludes, much like what I proposed for firmlinks in https://code.blicky.net/yorhel/ncdu/issues/153#issuecomment-966 - but I'll have to see to what extent that will work.

Yup, bind mounts are a pain and can't be detected with typical tooling. `du -x` and `find -xdev` etc will likely run into the same problem. I *think* this could be solved by parsing /proc/self/mounts and adding excludes, much like what I proposed for firmlinks in https://code.blicky.net/yorhel/ncdu/issues/153#issuecomment-966 - but I'll have to see to what extent that will work.

Alexis commented 2023年04月19日 09:13:17 +02:00 (Migrated from code.blicky.net)

1 - Could you send a donation link? You really saved us HOURS, and many headaches from our work! So I want donate something, anything, as a payback.
2 - I just saw the code. I thought how hard would would it be to implement it? I saw the files needed... but then again I read the 2.x announcement, and I read about Zig stability... XD sooo... It's a "No. Let's wait" situation.

1 - Could you send a donation link? You really saved us HOURS, and many headaches from our work! So I want donate something, anything, as a payback. 2 - I just saw the code. I thought how hard would would it be to implement it? I saw the files needed... but then again I read the 2.x announcement, and I read about Zig stability... XD sooo... It's a "No. Let's wait" situation.

Alexis commented 2023年04月19日 09:14:41 +02:00 (Migrated from code.blicky.net)

1 -> It's a thing of honour. My boss may not care to donate - I told him all the great result was due to your tool and me working a 20hour straight - but I believe I have to.
The very least, your tool made me happy for this time :D

1 -> It's a thing of honour. My boss may not care to donate - I told him all the great result was due to your tool and me working a 20hour straight - but I believe I have to. The very least, your tool made me happy for this time :D

yorhel commented 2023年04月19日 09:26:12 +02:00 (Migrated from code.blicky.net)

Thanks, your gesture is enough though, donations are too much of a hassle with taxes.

Thanks, your gesture is enough though, donations are too much of a hassle with taxes.

Alexis commented 2023年04月19日 09:40:51 +02:00 (Migrated from code.blicky.net)

:/
Okay

On another topic, is the main branch code evolving to be V. 2.x ? Or is it still 1.x
I don't see "Discussion" tab or something like that, nor a "send pm" here at code.blicky.net,
so excuse me asking here.

If it's the latest code, maybe I will do try to implement the mountpoints exclusion (of course you will have to harmonize it... I know not any Zig at all! It will be copy-paste and shallow searching till it works)

:/ Okay On another topic, is the main branch code evolving to be V. 2.x ? Or is it still 1.x I don't see "Discussion" tab or something like that, nor a "send pm" here at code.blicky.net, so excuse me asking here. If it's the latest code, maybe I will do try to implement the mountpoints exclusion (of course you will have to harmonize it... I know not any Zig at all! It will be copy-paste and shallow searching till it works)

yorhel commented 2023年04月19日 09:44:01 +02:00 (Migrated from code.blicky.net)

"zig" branch is for 2.x, "master" branch 1.x. Also check https://dev.yorhel.nl/contributing. I'm not even sure yet if parsing /proc/self/mounts is the way to go, so writing code at this point may be a little premature.

"zig" branch is for 2.x, "master" branch 1.x. Also check https://dev.yorhel.nl/contributing. I'm not even sure yet if parsing /proc/self/mounts is the way to go, so writing code at this point may be a little premature.

Alexis commented 2023年04月19日 10:28:37 +02:00 (Migrated from code.blicky.net)

I confirm in our system it sure has lines for many, if not all, mountpoints.

For example:

...
/dev/md2 /usr/share/cagefs-skeleton/var/lib/mysql ext4 rw,nosuid,relatime,stripe=256,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
/dev/md2 /usr/share/cagefs-skeleton/opt/cpanel/ea-php70/root/usr/bin ext4 ro,nosuid,relatime,stripe=256,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0
...

I confirm in our system it sure has lines for many, if not all, mountpoints. For example: ``` ... /dev/md2 /usr/share/cagefs-skeleton/var/lib/mysql ext4 rw,nosuid,relatime,stripe=256,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0 /dev/md2 /usr/share/cagefs-skeleton/opt/cpanel/ea-php70/root/usr/bin ext4 ro,nosuid,relatime,stripe=256,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0 ... ```

yorhel commented 2023年04月19日 11:16:00 +02:00 (Migrated from code.blicky.net)

Hmm, right, in https://code.blicky.net/yorhel/ncdu/issues/153 I assumed we could get both the source and destination paths of a bind mount, but /proc/self/mounts only gives us the device and the destination. Excluding all destinations is useful if the entire filesystem is being scanned, but can it give unintuitive results when scanning a subtree?

Imagine an fstab like:

/dev/something /mnt/something ext4 ..
/mnt/something/a /home/a none bind
/mnt/something/b /home/a/b none bind
/mnt/something/b/c /home/a/c none bind

That results in a mounts file like:

/dev/something /mnt/something
/dev/something /home/a
/dev/something /home/a/b
/dev/something /home/a/c

What should happen when scanning /home/a?

I'm also curious if and how mount namespaces affect the contents of the mounts file.

EDIT: And it may be worth checking if the format of the file has changed in kernel versions. And what happens with paths containing a tab, space or newline.

Hmm, right, in https://code.blicky.net/yorhel/ncdu/issues/153 I assumed we could get both the source and destination paths of a bind mount, but */proc/self/mounts* only gives us the device and the destination. Excluding all destinations is useful if the entire filesystem is being scanned, but can it give unintuitive results when scanning a subtree? Imagine an fstab like: ``` /dev/something /mnt/something ext4 .. /mnt/something/a /home/a none bind /mnt/something/b /home/a/b none bind /mnt/something/b/c /home/a/c none bind ``` That results in a mounts file like: ``` /dev/something /mnt/something /dev/something /home/a /dev/something /home/a/b /dev/something /home/a/c ``` What should happen when scanning */home/a*? I'm also curious if and how mount namespaces affect the contents of the mounts file. EDIT: And it may be worth checking if the format of the file has changed in kernel versions. And what happens with paths containing a tab, space or newline.

yorhel commented 2023年04月19日 17:14:50 +02:00 (Migrated from code.blicky.net)

Thinking about this more, I just realized that the solution I proposed for firmlinks don't generalize to bind mounts. Even with the source path available, in the above situation /mnt/something/b/c would still get scanned and counted twice.

I see only two correct solutions:

1. An option to exclude all bind mounts. Should be easier to implement and is probably useful, although there's still the challenge of figuring out which of the listed mounts for a particular device is the root and which are bind mounts.

2. Treat bind mounts as a shared resource, as if they were directory hardlinks. This seems significantly harder to implement.

Thinking about this more, I just realized that the solution I proposed for firmlinks don't generalize to bind mounts. Even with the source path available, in the above situation */mnt/something/b/c* would still get scanned and counted twice. I see only two correct solutions: 1. An option to exclude *all* bind mounts. Should be easier to implement and is probably useful, although there's still the challenge of figuring out which of the listed mounts for a particular device is the root and which are bind mounts. 2. Treat bind mounts as a shared resource, as if they were directory hardlinks. This seems significantly harder to implement.