Cygwin 3.6 possible issue handling compressed .pdb files on SSD?

Thu Jan 16 00:15:11 GMT 2025

On 2025年01月15日 10:26, Corinna Vinschen via Cygwin wrote:
> On Jan 15 09:12, Brian Inglis via Cygwin wrote:
>> On 2025年01月14日 03:13, Roland Mainz via Cygwin wrote:
>>> On Tue, Jan 14, 2025 at 7:19 AM Brian Inglis via Cygwin
>>> <cygwin@cygwin.com> wrote:
>>>> On 2025年01月13日 13:10, Roland Mainz via Cygwin wrote:
>>>>> I just hit an endless loop with /usr/bin/cp from "coreutils" version
>>>>> 9.5-1 copying a larger *.pdb file (it seems that only this specific
>>>>> file is affected...) from Visual Studio 19.
>>>>> [...]
>>> I think I found the problem:
>>> The *.pdb file uses NTFS compression:
>>> ---- snip ----
>>> /bin/winfsinfo filebasicinfo "$(cygpath -w
>>> $PWD/../build.vc19/x64/Debug/nfs41_driver.pdb)"
>>> (
>>> filename='C:\cygwin64\home\roland_mainz\work\msnfs41_uidmapping\ms-nfs41-client-kofemannvacation\build.vc19\x64\Debug\nfs41_driver.pdb'
>>> CreationTime=133812707624654816
>>> LastAccessTime=133813220892976366
>>> LastWriteTime=133812707639811081
>>> ChangeTime=133812707639811081
>>> typeset -a FileAttributes=(
>>> FILE_ATTRIBUTE_ARCHIVE
>>> FILE_ATTRIBUTE_COMPRESSED
>>> )
>>> )
>>> ---- snip ----
>>>>>> If I remove the "FILE_ATTRIBUTE_COMPRESSED" flag /bin/cp works without problems.
>>> I think the issues here are:
>>> 1. Coreutils 9.5-1 /bin/cp erroneously assumes that a file is sparse
>>> if the number of blocks is smaller than $((filesize / fs_blocksize)) -
>>> but in this case the file is NOT sparse, just compressed.
>>> 2. The loop to copy a sparse file is faulty because there are no holes
>>> in that file. That itself is IMHO already a bad idea to have a
>>> separate codepath for sparse files, just the normal codepath should
>>> use SEEK_HOLE and just skip those in the destination
>>>> A possible issue is that Cygwin assumes sparse files on SSD
>> No, that's not the problem, because SPARSE handling requires that
> the FILE_ATTRIBUTE_SPARSE_FILE flag of the file is actually set.
>> For instance, see lseek w/ SEEK_DATA/SEEK_HOLE, which handles
> non-sparse files as documented by the Linux man page
> https://man7.org/linux/man-pages/man2/lseek.2.html:
>> if (!has_attribute (FILE_ATTRIBUTE_SPARSE_FILE))
> {
> /* Default behaviour if sparse files are not supported:
> SEEK_DATA: seek to offset
> SEEK_HOLE: seek to EOF */
> fpi.CurrentByteOffset.QuadPart = (whence == SEEK_DATA)
> ? offset
> : fsi.EndOfFile.QuadPart;
> break;
> }
>>> so we need
>> fhandler/disk_file:fstat_helper to allow cp to handle compressed files
>> normally.
>> Say again? What exactly are you expecting from stat()? Fibbing about
> the number of blocks taken by a FS-compressed file?

Just a suggestion to work around the issue transparently?
> For the time being, it seems the assumption that #blocks * blocksize <
> filesize is not really correct.
>> Actually cp(1) should test if lseek(SEEK_HOLE) returns EOF. If so,
> the file isn't sparse.
>> On Cygwin it could also check with
>> (ioctl (fd, FS_IOC_GETFLAGS, &flags) & FS_SPARSE_FL) != 0
>> that the file is sparse, but the lseek test is target-independent.
>> On Cygwin and Linux you could also test with
>> (ioctl (fd, FS_IOC_GETFLAGS, &flags) & FS_COMPR_FL) != 0
>> that the file is compressed, but there's a problem so far. The
> Linux flag is called FS_COMPR_FL but the Cygwin flag is called
> FS_COMPRESSED_FL. THis flag is only supported on btrfs and I'm
> not sure it already existed when I added FS_IOC_GETFLAGS...
>> Anyway, I'll change FS_COMPR_FL to FS_COMPRESSED_FL and make
> FS_COMPRESSED_FL an alias for backward compat...

Thanks - I'll submit a bug report and ask what approach they would prefer?
-- 
Take care. Thanks, Brian Inglis Calgary, Alberta, Canada
La perfection est atteinte Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter not when there is no more to add
mais lorsqu'il n'y a plus rien à retrancher but when there is no more to cut
 -- Antoine de Saint-Exupéry