Weird issues on WSL2 #264

BratishkaErik commented 2025年07月04日 14:45:56 +02:00 (Migrated from code.blicky.net)

Hello, I have recently been integrating Gentoo WSL with my Windows 11 instance on notebook, and tried to use ncdu inside of powershell session for its intended purpose. I have noticed some bugs when testing:

1. ncdu seems to struggle with some files which inflate reported occupied space enormously compared to real data, for example it shows that I have 2.0 EiB directory on 300 GiB partition... (attachment 1).
2. This issue is reproducable on different partitions, see attachment 2 and 3. However, they seem to stuck sometimes when full scanning and stops responding to any keys, including "q", so I limit it there to just Users or Windows subfolders.
3. I have tried to use both 1-threaded and 8-threaded run, with and without --one-file-system, with and without --follow-symlinks, all with no luck. I can reproduce this on sys-fs/ncdu-2.8.2 Gentoo source build as well as on official binareis for 2.8.1 from your site.

This problem goes away as soon as I check file size using standard windows utilities or other programs like ls (both windows and linux versions). For example see attachment 4, 5.

I have run out of attachment slots, I'll continue with second message now.

Hello, I have recently been integrating Gentoo WSL with my Windows 11 instance on notebook, and tried to use ncdu inside of powershell session for its intended purpose. I have noticed some bugs when testing: 1) ncdu seems to struggle with some files which inflate reported occupied space enormously compared to real data, for example it shows that I have 2.0 EiB directory on 300 GiB partition... (attachment 1). 2) This issue is reproducable on different partitions, see attachment 2 and 3. However, they seem to stuck sometimes when full scanning and stops responding to any keys, including "q", so I limit it there to just Users or Windows subfolders. 3) I have tried to use both 1-threaded and 8-threaded run, with and without `--one-file-system`, with and without `--follow-symlinks`, all with no luck. I can reproduce this on `sys-fs/ncdu-2.8.2` Gentoo source build as well as on official binareis for 2.8.1 from your site. This problem goes away as soon as I check file size using standard windows utilities or other programs like `ls` (both windows and linux versions). For example see attachment 4, 5. I have run out of attachment slots, I'll continue with second message now.

BratishkaErik commented 2025年07月04日 14:50:51 +02:00 (Migrated from code.blicky.net)

wsl eza:
image

wsl ncdu after refreshing with different utilities:
image

I have not checked all instances yet, but they seem to be limited to small files from 0 to nearly 4 kb size as reported by Windows. It is not a serious bug in my case, since discrepancy is very obvious and can be considered just a small noise, but it still distracts somehow.

Interestingly, they also have more actual size than size on disk? I don't want to attach untranslated screenshot, so I will sent this message, change system language and relogin.

`wsl eza`:  `wsl ncdu` after refreshing with different utilities:  I have not checked all instances yet, but they seem to be limited to small files from 0 to nearly 4 kb size as reported by Windows. It is not a serious bug in my case, since discrepancy is very obvious and can be considered just a small noise, but it still distracts somehow. Interestingly, they also have more actual size than size on disk? I don't want to attach untranslated screenshot, so I will sent this message, change system language and relogin.

BratishkaErik commented 2025年07月04日 14:55:43 +02:00 (Migrated from code.blicky.net)

Example of normal file:
image

And faulty one (in ncdu case reported as 10 EiB):
image
Снимок экрана 2025年07月04日 170537

I should probably also note that problem reproduces when I'm running not from Windows PowerShell side, but Gentoo (root) bash too

This is final attachment for now, let me know if you need something more.

Example of normal file:  And faulty one (in ncdu case reported as 10 EiB):   I should probably also note that problem reproduces when I'm running not from Windows PowerShell side, but Gentoo (root) bash too This is final attachment for now, let me know if you need something more.

BratishkaErik commented 2025年07月04日 15:06:04 +02:00 (Migrated from code.blicky.net)

Also, this is reproducible in ncdu 1.21 too:
image
image

Explorer data:
image
image

Also, this is reproducible in ncdu 1.21 too:   Explorer data:  

yorhel commented 2025年07月04日 15:39:51 +02:00 (Migrated from code.blicky.net)

Ncdu relies on stat() to provide sensible data. If WSL1 doesn't implement that usefully or is buggy, then the numbers will obviously be wrong.

This problem goes away as soon as I check file size using standard windows utilities or other programs like ls (both windows and linux versions).

Odd. Even with Gentoo's coreutils ls? How about stat $file? What is that doing differently?

Ncdu relies on `stat()` to provide sensible data. If WSL1 doesn't implement that usefully or is buggy, then the numbers will obviously be wrong. > This problem goes away as soon as I check file size using standard windows utilities or other programs like `ls` (both windows and linux versions). Odd. Even with Gentoo's coreutils `ls`? How about `stat $file`? What is that doing differently?

BratishkaErik commented 2025年07月04日 15:43:27 +02:00 (Migrated from code.blicky.net)

Ncdu relies on stat() to provide sensible data. If WSL1 doesn't implement that usefully or is buggy, then the numbers will obviously be wrong.

I'm using WSL2 (it's in the title), so this sits in Hyper-V VM. Maybe this quote will help:

WSL 2 is the default distro type when installing a Linux distribution. WSL 2 uses virtualization technology to run a Linux kernel inside of a lightweight utility virtual machine (VM). Linux distributions run as isolated containers inside of the WSL 2 managed VM. Linux distributions running via WSL 2 will share the same network namespace, device tree (other than /dev/pts), CPU/Kernel/Memory/Swap, /init binary, but have their own PID namespace, Mount namespace, User namespace, Cgroup namespace, and init process.

This problem goes away as soon as I check file size using standard windows utilities or other programs like ls (both windows and linux versions).

Odd. Even with Gentoo's coreutils ls?

Yes, with both coreutils ls, rust eza, and powershell ls a.k.a Get-ChildItem.

> Ncdu relies on `stat()` to provide sensible data. If WSL1 doesn't implement that usefully or is buggy, then the numbers will obviously be wrong. I'm using WSL2 (it's in the title), so this sits in Hyper-V VM. Maybe this quote will help: > WSL 2 is the default distro type when installing a Linux distribution. WSL 2 uses virtualization technology to run a Linux kernel inside of a lightweight utility virtual machine (VM). Linux distributions run as isolated containers inside of the WSL 2 managed VM. Linux distributions running via WSL 2 will share the same network namespace, device tree (other than `/dev/pts`), CPU/Kernel/Memory/Swap, `/init` binary, but have their own PID namespace, Mount namespace, User namespace, Cgroup namespace, and `init` process. > > > This problem goes away as soon as I check file size using standard windows utilities or other programs like `ls` (both windows and linux versions). > > Odd. Even with Gentoo's coreutils `ls`? Yes, with both coreutils `ls`, rust `eza`, and powershell `ls` a.k.a `Get-ChildItem`.

BratishkaErik commented 2025年07月04日 15:50:03 +02:00 (Migrated from code.blicky.net)

Example:
image
image

Example:  

yorhel commented 2025年07月04日 15:51:03 +02:00 (Migrated from code.blicky.net)

WSL2, so it's running an actual Linux kernel? I suppose in that case we should be able to strace. Could you run a scan under strace --no-abbrev -o logfile -e newfstatat ncdu and find the respective stat call on a file that has the wrong size reported?
("newfstatat" might just be "fstatat" or something else, I'm not familiar with Linux syscall naming)

WSL2, so it's running an actual Linux kernel? I suppose in that case we should be able to strace. Could you run a scan under `strace --no-abbrev -o logfile -e newfstatat ncdu` and find the respective stat call on a file that has the wrong size reported? ("newfstatat" might just be "fstatat" or something else, I'm not familiar with Linux syscall naming)

yorhel commented 2025年07月04日 15:52:13 +02:00 (Migrated from code.blicky.net)

Note in your screenshot that the "apparent size" looks to be correct, it's the "disk usage" that's wrong. You can compare that with coreutils du or stat.

Note in your screenshot that the "apparent size" looks to be correct, it's the "disk usage" that's wrong. You can compare that with coreutils `du` or `stat`.

BratishkaErik commented 2025年07月04日 16:01:50 +02:00 (Migrated from code.blicky.net)

I don't know what happened, but first time du -hs resetted file sizes:
image

But later it did not:
image

ls and company still succeeded in "size correction".

strace --no-abbrev -o logfile -e newfstatat ncdu

I'm not proficient in strace but it looks like there is not enough data collected:

newfstatat(AT_FDCWD, "/mnt/e/Users/LEGION/AppData/Local/Microsoft/OneDrive/logs/Personal", {st_dev=makedev(0, 0x48), st_ino=281474976827623, st_mode=S_IFDIR|0777, st_nlink=1, st_uid=1000, st_gid=1000, st_blksize=4096, st_blocks=0, st_size=4096, st_atime=1751637541 /* 2025年07月04日T18:59:01.456866700+0500 */, st_atime_nsec=456866700, st_mtime=1726711460 /* 2024年09月19日T07:04:20.117448600+0500 */, st_mtime_nsec=117448600, st_ctime=1750358424 /* 2025年06月19日T23:40:24.311790600+0500 */, st_ctime_nsec=311790600}, 0) = 0
newfstatat(AT_FDCWD, "/home/bratishkaerik/.terminfo", 0x608e761f1ea0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/etc/terminfo", 0x608e761f1ea0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/share/terminfo", {st_dev=makedev(0x8, 0x30), st_ino=53768, st_mode=S_IFDIR|0755, st_nlink=44, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=8, st_size=4096, st_atime=1751630387 /* 2025年07月04日T16:59:47.249957075+0500 */, st_atime_nsec=249957075, st_mtime=1747630870 /* 2025年05月19日T10:01:10.530645208+0500 */, st_mtime_nsec=530645208, st_ctime=1750356350 /* 2025年06月19日T23:05:50.678923316+0500 */, st_ctime_nsec=678923316}, 0) = 0
+++ exited with 0 +++

I don't know what happened, but first time `du -hs` resetted file sizes:  But later it did not:  ls and company still succeeded in "size correction". > strace --no-abbrev -o logfile -e newfstatat ncdu I'm not proficient in strace but it looks like there is not enough data collected: ``` newfstatat(AT_FDCWD, "/mnt/e/Users/LEGION/AppData/Local/Microsoft/OneDrive/logs/Personal", {st_dev=makedev(0, 0x48), st_ino=281474976827623, st_mode=S_IFDIR|0777, st_nlink=1, st_uid=1000, st_gid=1000, st_blksize=4096, st_blocks=0, st_size=4096, st_atime=1751637541 /* 2025年07月04日T18:59:01.456866700+0500 */, st_atime_nsec=456866700, st_mtime=1726711460 /* 2024年09月19日T07:04:20.117448600+0500 */, st_mtime_nsec=117448600, st_ctime=1750358424 /* 2025年06月19日T23:40:24.311790600+0500 */, st_ctime_nsec=311790600}, 0) = 0 newfstatat(AT_FDCWD, "/home/bratishkaerik/.terminfo", 0x608e761f1ea0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/etc/terminfo", 0x608e761f1ea0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/usr/share/terminfo", {st_dev=makedev(0x8, 0x30), st_ino=53768, st_mode=S_IFDIR|0755, st_nlink=44, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=8, st_size=4096, st_atime=1751630387 /* 2025年07月04日T16:59:47.249957075+0500 */, st_atime_nsec=249957075, st_mtime=1747630870 /* 2025年05月19日T10:01:10.530645208+0500 */, st_mtime_nsec=530645208, st_ctime=1750356350 /* 2025年06月19日T23:05:50.678923316+0500 */, st_ctime_nsec=678923316}, 0) = 0 +++ exited with 0 +++ ```

BratishkaErik commented 2025年07月04日 16:05:30 +02:00 (Migrated from code.blicky.net)

I can send full file with strace --no-abbrev -o logfile ncdu '/mnt/e/Users/LEGION/AppData/Local/Microsoft/OneDrive/logs/Personal/' instead if you want, it's not big, 328 lines:
logfile

I can send full file with `strace --no-abbrev -o logfile ncdu '/mnt/e/Users/LEGION/AppData/Local/Microsoft/OneDrive/logs/Personal/'` instead if you want, it's not big, 328 lines: [logfile](/attachments/2e55038e-ac15-43a1-8cb5-028192b8d2fb)

yorhel commented 2025年07月04日 16:53:41 +02:00 (Migrated from code.blicky.net)

Hmm, that strace might have needed an -f to log the threads as well. Anyway, your du and stat screenshots confirm that it's not ncdu - Windows indeed seems to be providing bogus data... sometimes.

Good luck digging into why that happens. :)

Hmm, that strace might have needed an `-f` to log the threads as well. Anyway, your du and stat screenshots confirm that it's not ncdu - Windows indeed seems to be providing bogus data... sometimes. Good luck digging into why that happens. :)

BratishkaErik commented 2025年07月04日 17:08:40 +02:00 (Migrated from code.blicky.net)

strace --no-abbrev -o logfile ncdu -0 -t1 --ignore-config --color off '/mnt/d/Windows/Servicing/Sessions/'

I think I found the cause:

newfstatat(3, "31187475_3153972839.xml", {st_dev=makedev(0, 0x47), st_ino=281474976847340, st_mode=S_IFREG|0555, st_nlink=1, st_uid=1000, st_gid=1000, st_blksize=4096, st_blocks=4564312022170035, st_size=558, st_atime=1751633329 /* 2025年07月04日T17:48:49.720310100+0500 */, st_atime_nsec=720310100, st_mtime=1750445232 /* 2025年06月20日T23:47:12.967250200+0500 */, st_mtime_nsec=967250200, st_ctime=1750445232 /* 2025年06月20日T23:47:12.967250200+0500 */, st_ctime_nsec=967250200}, AT_SYMLINK_NOFOLLOW) = 0

st_size and st_blocks differ greatly, with this I had found relevant WSL issue easily:
https://github.com/Microsoft/WSL/issues/2844

Some quotes for historical preservance:

Small files can be embedded directly in the MFT so blkcnt_t should be able to be zero even if the file is non-empty. A zero byte and one byte file takes up exactly the same amount of space on NTFS, but ntfs-3g report the former as zero blocks and the latter as one block. Arguably ntfs-3g is the one that is wrong here, but they might have done this to work around buggy software.
Really the IO libraries in question should fix this, the man page even points out not to rely on it:

Use of the st_blocks and st_blksize fields may be less portable. (They were introduced in BSD. The interpretation differs between systems, and possibly on a single system when NFS mounts are involved.)

https://manned.org/man/stat.2#head8

I guess Windows is therefore theoretically fully compliant?

`strace --no-abbrev -o logfile ncdu -0 -t1 --ignore-config --color off '/mnt/d/Windows/Servicing/Sessions/'` I think I found the cause: ``` newfstatat(3, "31187475_3153972839.xml", {st_dev=makedev(0, 0x47), st_ino=281474976847340, st_mode=S_IFREG|0555, st_nlink=1, st_uid=1000, st_gid=1000, st_blksize=4096, st_blocks=4564312022170035, st_size=558, st_atime=1751633329 /* 2025年07月04日T17:48:49.720310100+0500 */, st_atime_nsec=720310100, st_mtime=1750445232 /* 2025年06月20日T23:47:12.967250200+0500 */, st_mtime_nsec=967250200, st_ctime=1750445232 /* 2025年06月20日T23:47:12.967250200+0500 */, st_ctime_nsec=967250200}, AT_SYMLINK_NOFOLLOW) = 0 ``` `st_size` and `st_blocks` differ greatly, with this I had found relevant WSL issue easily: https://github.com/Microsoft/WSL/issues/2844 Some quotes for historical preservance: > Small files can be embedded directly in the MFT so blkcnt_t should be able to be zero even if the file is non-empty. A zero byte and one byte file takes up exactly the same amount of space on NTFS, but ntfs-3g report the former as zero blocks and the latter as one block. Arguably ntfs-3g is the one that is wrong here, but they might have done this to work around buggy software. > Really the IO libraries in question should fix this, the man page even points out not to rely on it: > > Use of the _st_blocks_ and _st_blksize_ fields may be less portable. (They were introduced in BSD. The interpretation differs between systems, and possibly on a single system when NFS mounts are involved.) https://manned.org/man/stat.2#head8 I guess Windows is therefore theoretically fully compliant?

yorhel commented 2025年07月04日 17:31:57 +02:00 (Migrated from code.blicky.net)

The linked issue only discusses the possibility of st_blocks being zero, but what we're seeing here is a complete nonsense value. I doubt that's working as intended...

The linked issue only discusses the possibility of `st_blocks` being zero, but what we're seeing here is a complete nonsense value. I doubt that's working as intended...

BratishkaErik commented 2025年07月04日 17:38:16 +02:00 (Migrated from code.blicky.net)

The linked issue only discusses the possibility of st_blocks being zero, but what we're seeing here is a complete nonsense value. I doubt that's working as intended...

I just rechecked, they are also apparently on WSL1 and not WSL2, which should emulate Linux kernel as close as possible...

Also: ls and eza use here statx instead of stat, and they return correct values:

statx(AT_FDCWD, "/mnt/d/Windows/Servicing/Sessions/31187475_3153972839.xml", AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW|AT_NO_AUTOMOUNT, STATX_MODE|STATX_NLINK|STATX_GID|STATX_MTIME|STATX_SIZE, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_blksize=4096, stx_attributes=0, stx_nlink=1, stx_uid=1000, stx_gid=1000, stx_mode=S_IFREG|0555, stx_ino=281474976847340, stx_size=558, stx_blocks=0, stx_attributes_mask=STATX_ATTR_AUTOMOUNT|STATX_ATTR_MOUNT_ROOT|STATX_ATTR_DAX, stx_atime={tv_sec=1751633329, tv_nsec=720310100} /* 2025年07月04日T17:48:49.720310100+0500 */, stx_ctime={tv_sec=1750445232, tv_nsec=967250200} /* 2025年06月20日T23:47:12.967250200+0500 */, stx_mtime={tv_sec=1750445232, tv_nsec=967250200} /* 2025年06月20日T23:47:12.967250200+0500 */, stx_rdev_major=0, stx_rdev_minor=0, stx_dev_major=0, stx_dev_minor=71, stx_mnt_id=0x85}) = 0

I guess sequence of stat -> statx -> stat produces my result of "1 incorrect result, statx triggers something, 2 correct results"?

> The linked issue only discusses the possibility of `st_blocks` being zero, but what we're seeing here is a complete nonsense value. I doubt that's working as intended... I just rechecked, they are also apparently on WSL1 and not WSL2, which should emulate Linux kernel as close as possible... Also: `ls` and `eza` use here `statx` instead of `stat`, and they return correct values: ``` statx(AT_FDCWD, "/mnt/d/Windows/Servicing/Sessions/31187475_3153972839.xml", AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW|AT_NO_AUTOMOUNT, STATX_MODE|STATX_NLINK|STATX_GID|STATX_MTIME|STATX_SIZE, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_blksize=4096, stx_attributes=0, stx_nlink=1, stx_uid=1000, stx_gid=1000, stx_mode=S_IFREG|0555, stx_ino=281474976847340, stx_size=558, stx_blocks=0, stx_attributes_mask=STATX_ATTR_AUTOMOUNT|STATX_ATTR_MOUNT_ROOT|STATX_ATTR_DAX, stx_atime={tv_sec=1751633329, tv_nsec=720310100} /* 2025年07月04日T17:48:49.720310100+0500 */, stx_ctime={tv_sec=1750445232, tv_nsec=967250200} /* 2025年06月20日T23:47:12.967250200+0500 */, stx_mtime={tv_sec=1750445232, tv_nsec=967250200} /* 2025年06月20日T23:47:12.967250200+0500 */, stx_rdev_major=0, stx_rdev_minor=0, stx_dev_major=0, stx_dev_minor=71, stx_mnt_id=0x85}) = 0 ``` I guess sequence of `stat` -> `statx` -> `stat` produces my result of "1 incorrect result, statx triggers something, 2 correct results"?

yorhel commented 2025年07月04日 17:44:23 +02:00 (Migrated from code.blicky.net)

That statx() call does not include the STATX_BLOCKS flag, so that field is left empty. But that's an interesting thing to test: can you try strace on ls -s? (-s displays the block count, may be a GNU extension)

That `statx()` call does not include the `STATX_BLOCKS` flag, so that field is left empty. But that's an interesting thing to test: can you try strace on `ls -s`? (`-s` displays the block count, may be a GNU extension)

BratishkaErik commented 2025年07月04日 17:47:46 +02:00 (Migrated from code.blicky.net)

@yorhel wrote in https://code.blicky.net/yorhel/ncdu/issues/264#issuecomment-1741:

That statx() call does not include the STATX_BLOCKS flag, so that field is left empty.

stat also says (after statx) that blocks are zero, so I think it is intended:
image

But that's an interesting thing to test: can you try strace on ls -s? (-s displays the block count, may be a GNU extension)

0 /mnt/d/Windows/Servicing/Sessions/31187475_3153972839.xml

logfile

@yorhel wrote in https://code.blicky.net/yorhel/ncdu/issues/264#issuecomment-1741: > That `statx()` call does not include the `STATX_BLOCKS` flag, so that field is left empty. stat also says (after statx) that blocks are zero, so I think it is intended:  > But that's an interesting thing to test: can you try strace on ls -s? (-s displays the block count, may be a GNU extension) ``` 0 /mnt/d/Windows/Servicing/Sessions/31187475_3153972839.xml ``` [logfile](/attachments/538ba83f-002f-4c35-a345-35b5bffdd9c6)

yorhel commented 2025年07月04日 17:56:16 +02:00 (Migrated from code.blicky.net)

Interesting, that looks good. Can you try the stat command again? I just checked and on my system that uses statx() as well. Given that the error only happens sometimes, it's hard to tell whether the difference in system call is what causes the issue or if this is just a randomly correct answer.

Interesting, that looks good. Can you try the `stat` command again? I just checked and on my system that uses `statx()` as well. Given that the error only happens sometimes, it's hard to tell whether the difference in system call is what causes the issue or if this is just a randomly correct answer.

BratishkaErik commented 2025年07月04日 18:37:27 +02:00 (Migrated from code.blicky.net)

Interesting, that looks good. Can you try the stat command again? I just checked and on my system that uses statx() as well. Given that the error only happens sometimes, it's hard to tell whether the difference in system call is what causes the issue or if this is just a randomly correct answer.

Sorry, my desktop is occupied now so I cant check for now. I read coreutils github mirror and it seems like stat command does use statx when available.

Also I ran out of anomalies to compare, will wait for some bit, maybe some time of interactions will bring some of them back.

Maybe some very simple stat-like C/Zig program with option to choose between stat and statx exist, so that it can be run in some bash loop or whatever.

> Interesting, that looks good. Can you try the `stat` command again? I just checked and on my system that uses `statx()` as well. Given that the error only happens sometimes, it's hard to tell whether the difference in system call is what causes the issue or if this is just a randomly correct answer. Sorry, my desktop is occupied now so I cant check for now. I read coreutils github mirror and it seems like stat command does use `statx` when available. Also I ran out of anomalies to compare, will wait for some bit, maybe some time of interactions will bring some of them back. Maybe some very simple stat-like C/Zig program with option to choose between stat and statx exist, so that it can be run in some bash loop or whatever.

BratishkaErik commented 2025年07月04日 22:58:20 +02:00 (Migrated from code.blicky.net)

For some reason ls -s this time did not refresh, but ls -lags did...
image
image
image

For some reason `ls -s` this time did not refresh, but `ls -lags` did...   

yorhel commented 2025年07月15日 18:30:25 +02:00 (Migrated from code.blicky.net)

Closing for now, doesn't seem like something we can sensibly work around within ncdu.

Closing for now, doesn't seem like something we can sensibly work around within ncdu.