extensions: Use system clang, if it supports the wasm target #64

selfisekai commented 2026年03月08日 05:38:20 +01:00

Bug: #58

Bug: https://codeberg.org/GramEditor/gram/issues/58

tjk commented 2026年03月22日 21:43:55 +01:00

With this change, what would be the normal way to get the WASI SDK for users with clang installed?

My distro (Fedora) provides clang but not the WASI SDK. Clang successfully runs the test command (clang --target=wasm32-wasip2 -print-supported-cpus), so this change uses it, but I don't have the WASI SDK commands needed to build extensions, e.g. wasm-component-ld and wasm-ld, so builds fail. I downloaded and installed WASI SDK to the location Gram would put it if clang weren't installed, but it's not used because this supersedes it. I could only get an extension to build by reverting this change.

(The extension doesn't seem to take effect, but I'm tracking that down separately.)

For reference, my rustup target install of wasm32-wasip2 has the wasm-* commands, but even if I add them to my PATH, they're not compatible with system-installed clang.

With this change, what would be the normal way to get the WASI SDK for users with clang installed? My distro (Fedora) provides clang but not the WASI SDK. Clang successfully runs the test command (`clang --target=wasm32-wasip2 -print-supported-cpus`), so this change uses it, but I don't have the WASI SDK commands needed to build extensions, e.g. wasm-component-ld and wasm-ld, so builds fail. I downloaded and installed WASI SDK to the location Gram would put it if clang weren't installed, but it's not used because this supersedes it. I could only get an extension to build by reverting this change. (The extension doesn't seem to take effect, but I'm tracking that down separately.) For reference, my rustup target install of wasm32-wasip2 has the wasm-* commands, but even if I add them to my PATH, they're not compatible with system-installed clang.

krig commented 2026年03月22日 21:47:35 +01:00

@tjk wrote in #64 (comment):

With this change, what would be the normal way to get the WASI SDK for users with clang installed?

My distro (Fedora) provides clang but not the WASI SDK. Clang successfully runs the test command (clang --target=wasm32-wasip2 -print-supported-cpus), so this change uses it, but I don't have the WASI SDK commands needed to build extensions, e.g. wasm-component-ld and wasm-ld, so builds fail. I downloaded and installed WASI SDK to the location Gram would put it if clang weren't installed, but it's not used because this supersedes it. I could only get an extension to build by reverting this change.

(The extension doesn't seem to take effect, but I'm tracking that down separately.)

For reference, my rustup target install of wasm32-wasip2 has the wasm-* commands, but even if I add them to my PATH, they're not compatible with system-installed clang.

Ah. That's a problem, then. We should at least make sure that the wasm-* commands are also available, but I wonder if there is any way to ensure that they actually work with the system clang.

@tjk wrote in https://codeberg.org/GramEditor/gram/pulls/64#issuecomment-11940873: > With this change, what would be the normal way to get the WASI SDK for users with clang installed? > > My distro (Fedora) provides clang but not the WASI SDK. Clang successfully runs the test command (`clang --target=wasm32-wasip2 -print-supported-cpus`), so this change uses it, but I don't have the WASI SDK commands needed to build extensions, e.g. wasm-component-ld and wasm-ld, so builds fail. I downloaded and installed WASI SDK to the location Gram would put it if clang weren't installed, but it's not used because this supersedes it. I could only get an extension to build by reverting this change. > > (The extension doesn't seem to take effect, but I'm tracking that down separately.) > > For reference, my rustup target install of wasm32-wasip2 has the wasm-* commands, but even if I add them to my PATH, they're not compatible with system-installed clang. Ah. That's a problem, then. We should at least make sure that the wasm-* commands are also available, but I wonder if there is any way to ensure that they actually work with the system clang.

krig commented 2026年03月22日 21:59:35 +01:00

@tjk What do you see in the logs if you try to install with wasm-ld from WASI SDK?

@tjk What do you see in the logs if you try to install with `wasm-ld` from WASI SDK?

krig commented 2026年03月22日 22:00:13 +01:00

I pushed a fix to at least make sure that wasm-ld and wasm-component-ld are available.

I pushed a fix to at least make sure that `wasm-ld` and `wasm-component-ld` are available.

tjk commented 2026年03月22日 22:14:28 +01:00

What do you see in the logs if you try to install with wasm-ld from WASI SDK?

Do you mean in the case where I've reverted this change? That's the only way the WASI SDK is used. In that case, it builds and links correctly, and I wind up with the WASM artifact in my Gram cache dir as expected. I imagine those logs wouldn't be very interesting? (As I mentioned earlier, the extension doesn't seem to activate / work, but I'm assuming that's a separate issue.)

If I don't revert this change, and I add -v to the clang call to see how it fails, it first can't find the wasm-component-ld binary; if I add that to my path from rustup, then it can't find wasm-ld; if I add that too, then I get a more complicated error that led me to believe they weren't compatible with my system's clang.

It sounds like your new commit would fix the second case, because I had to do unusual things to add in the incompatible wasm commands - most people wouldn't do that, so it wouldn't try to use system clang if it didn't also have the WASI SDK, and then Gram would correctly set up its own WASI SDK.

For which scenario would the logs be helpful? I'm happy to test and paste logs when I understand :)

> What do you see in the logs if you try to install with `wasm-ld` from WASI SDK? Do you mean in the case where I've reverted this change? That's the only way the WASI SDK is used. In that case, it builds and links correctly, and I wind up with the WASM artifact in my Gram cache dir as expected. I imagine those logs wouldn't be very interesting? (As I mentioned earlier, the extension doesn't seem to activate / work, but I'm assuming that's a separate issue.) If I don't revert this change, and I add `-v` to the clang call to see how it fails, it first can't find the wasm-component-ld binary; if I add that to my path from rustup, then it can't find wasm-ld; if I add that too, then I get a more complicated error that led me to believe they weren't compatible with my system's clang. It sounds like your new commit would fix the second case, because I had to do unusual things to add in the incompatible wasm commands - most people wouldn't do that, so it wouldn't try to use system clang if it didn't also have the WASI SDK, and then Gram would correctly set up its own WASI SDK. For which scenario would the logs be helpful? I'm happy to test and paste logs when I understand :)

krig commented 2026年03月22日 22:49:54 +01:00

For which scenario would the logs be helpful? I'm happy to test and paste logs when I understand :)

I think you are right that this should solve it for most people, but I was wondering if there was some easy way that I could determine if the wasm-ld installed on the system would be compatible with the clang version that's installed.

> For which scenario would the logs be helpful? I'm happy to test and paste logs when I understand :) I think you are right that this should solve it for most people, but I was wondering if there was some easy way that I could determine if the wasm-ld installed on the system would be compatible with the clang version that's installed.

tjk commented 2026年03月23日 00:27:41 +01:00

I can confirm that that fixed it! Thank you.

I also figured out my issue with the built extension not working. I had downloaded the latest WASI SDK version, 32. Everything seemed to be OK with the build, but the extension didn't load, giving the following error:

2026年03月22日T16:20:47-07:00 ERROR [language::language_registry] failed to load language Fish:
Failed to instantiate Wasm module: invalid import '__wasi_init_tp'

With your new change that lets Gram install the WASI SDK, it picks version 25, and now the extension loads and works fine. I had no specific reason to use a newer version, so this is fine by me. It might be worth figuring out someday in case there's a security-related update that could let extensions do bad things, though.

I can confirm that that fixed it! Thank you. I also figured out my issue with the built extension not working. I had downloaded the latest WASI SDK version, 32. Everything seemed to be OK with the build, but the extension didn't load, giving the following error: ```plain 2026年03月22日T16:20:47-07:00 ERROR [language::language_registry] failed to load language Fish: Failed to instantiate Wasm module: invalid import '__wasi_init_tp' ``` With your new change that lets Gram install the WASI SDK, it picks version 25, and now the extension loads and works fine. I had no specific reason to use a newer version, so this is fine by me. It might be worth figuring out someday in case there's a security-related update that could let extensions do bad things, though.

krig commented 2026年03月23日 09:08:41 +01:00

Ah, yes, makes sense. I wonder how Zed means to handle this, they will either need to break all extensions or maintain compatibility with both new and old WASI versions. For now it seems fine to stay on the old version.

My plan is still to move away from WASI and maybe pull in lua as an extension language, we'll see.

Ah, yes, makes sense. I wonder how Zed means to handle this, they will either need to break all extensions or maintain compatibility with both new and old WASI versions. For now it seems fine to stay on the old version. My plan is still to move away from WASI and maybe pull in lua as an extension language, we'll see.

selfisekai commented 2026年03月24日 06:15:21 +01:00

I think while a new extension system would be much work, throwing away tree-sitter is just unrealistic

I think while a new extension system would be much work, throwing away tree-sitter is just unrealistic

krig commented 2026年03月24日 07:55:54 +01:00

yeah, tree-sitter would have to stay. But compiling tree-sitter grammars natively is much easier than compiling to Wasm / WASI.

yeah, tree-sitter would have to stay. But compiling tree-sitter grammars natively is much easier than compiling to Wasm / WASI.