The first commit in this PR introduces a testcase for sketches with (working or broken) symlinks, but the "working symlink" case seems to fail to compile on Windows somehow (I tested just the first commit, which seems to create the symlink successfully given the lack of an exception, but then produces a link error that suggests the symlink is not included in the build). Maybe the symlink is created but the subsequent opening of it fails, which causes it to be silently ignored? If so, then a testrun of the full PR should show an error. We'll know once I submit this PR and the integration tests complete :-)

After fixing two more small issues with the testcase, the github test run reveals the problem:

E + Error during build: Can't open sketch: open C:\Users\runneradmin\AppData\Local\Temp\tmp3uppf06d\CompileIntegrationTestSymlinkOk\link.ino: The process cannot access the file because it is being used by another process.

It turns out that the NamedTemporyFile I used to create a target for the working symlink necessarily keeps the file opened (since it will be deleted on close), but on Windows that prevents opening it a second time (due to the O_TEMPORARY flag used). This is considered a bug in Python, but as of yet still unfixed.

I've pushed another commit that should fix the testcase, by passing delete=False to NamedTemporaryFile, which allows closing the file after writing to it. To ensure it is still deleted, I added a try/finally block to delete the file.

Integration tests are failing as of now because the CLI fails to install adafruit:samd tries to install an ARM tool from an URL with an bad certificate. Don't think we can do much about it other than wait.

$ arduino-cli core install adafruit:samd@1.6.4 
Downloading packages...
Error during install: Error downloading tool adafruit:arm-none-eabi-gcc@9-2019q4: Get "https://developer.arm.com/-/media/Files/downloads/gnu-rm/9-2019q4/gcc-arm-none-eabi-9-2019-q4-major-x86_64-linux.tar.bz2": x509: certificate signed by unknown authority
``

Am not sure the arduino-cli should handle this case. My main concern is the fact that the issue you're having could be solved in a different way by calling compile with the --export-binaries flag so that binaries are exported build/<FQBN> and let clangd read it from there.
I wouldn't make this big of change in the CLI to solve an issue that only requires some more configuration. 😕

Integration tests are failing as of now because the CLI fails to install adafruit:samd tries to install an ARM tool from an URL with an bad certificate. Don't think we can do much about it other than wait.

Heh. In any case, I think I did see the Windows integration tests pass at some point (but then I pushed another comment to fix some python style issue), so I think the integration part of the this is now good.

My main concern is the fact that the issue you're having could be solved in a different way by calling compile with the --export-binaries flag so that binaries are exported build/ and let clangd read it from there.
I wouldn't make this big of change in the CLI to solve an issue that only requires some more configuration. confused

Yeah, I also looked at a solution like that, but that requires explicitly passing the path the compilation_commands.json to clangd, which seems tricky at least in my environment (I'm using YouCompleteMe in vim, which I think also looks at compile_commands.json, but is not so configurable in this regard). I could look closer at that, though.

However, note that even when the last commit of this PR is left out (the compile_commands.json exception), I think the other changes made have merit, so I think it would still be worth discussing those (and as mentioned, maybe not the exact implementation, but the behavior, and the testcases are also already fine like this).

Heh. In any case, I think I did see the Windows integration tests pass at some point (but then I pushed another comment to fix some python style issue), so I think the integration part of the this is now good.

Should be fine now, I already rerunned the failing jobs.

Yeah, I also looked at a solution like that, but that requires explicitly passing the path the compilation_commands.json to clangd, which seems tricky at least in my environment (I'm using YouCompleteMe in vim, which I think also looks at compile_commands.json, but is not so configurable in this regard). I could look closer at that, though.

I think you can do it something like this:

let g:ycm_compilation_database_folder = 'build'

From here: ycm-core/YouCompleteMe#2741 (comment)

I didn't test it, I don't use YCM on Vim.

However, note that even when the last commit of this PR is left out (the compile_commands.json exception), I think the other changes made have merit, so I think it would still be worth discussing those (and as mentioned, maybe not the exact implementation, but the behavior, and the testcases are also already fine like this).

Yeah, the other changes are interesting but am still thinking if I like the changes in arduino/go-paths-helper#13, probably I'd use a different approach. 🤔

Seems I forgot to reply here, I was prompted by #1765 to have a look again here.

I think you can do it something like this:

Seems like that approach would indeed work for YouCompleteMe in vim, but does require a bunch of custom python code, so is very specific to YCM (i.e. it's not just a matter of passing some option to clangd) and unlikely to be easily supported in other environments too.

Regardless of the particulars of the compilation_commands.json usecase, the issue is maybe more generic: Under what circumstances should a broken symlink affect the build? It seems #1765 is essentially this same problem, but triggered by a symlink created by git (or the user, not entirely sure).