Remove compression and reduce memory footprint #79

jacobsandlund commented 2025年07月07日 02:33:45 +02:00

This PR removes the compression during codegen and instead writes out zig files that can be imported in src, so that the initializing of all modules is now (with one exception) just setting pointers to point into constant data space. Additionally, a few modules required 0x11000 length arrays, and have now been changed to two-tier data structures (using something like 65-75 kB instead of ~4-8 MB).

This PR keeps the same(*) API, so that anyone wanting these performance gains doesn't need breaking changes. That said, I'll be PR'ing a change to the API separately (after a few of these PRs are merged, since the changes are stacked on top of them, on my branch), as now all methods can just use constant data instead of runtime data.

Note that I tested this change in a real world application, and compression was only saving 16 bytes in total for 3 modules.

*: CompatData.toNfkd now returns []const u21 instead of []u21

This PR removes the compression during codegen and instead writes out zig files that can be imported in `src`, so that the initializing of all modules is now (with one exception) just setting pointers to point into constant data space. Additionally, a few modules required 0x11000 length arrays, and have now been changed to two-tier data structures (using something like 65-75 kB instead of ~4-8 MB). This PR keeps the same(*) API, so that anyone wanting these performance gains doesn't need breaking changes. That said, I'll be PR'ing a change to the API separately (after a few of these PRs are merged, since the changes are stacked on top of them, [on my branch](https://codeberg.org/jacobsandlund/zg/src/branch/jacob)), as now all methods can just use constant data instead of runtime data. Note that I tested this change in a real world application, and compression was only saving 16 bytes in total for 3 modules. *: `CompatData.toNfkd` now returns `[]const u21` instead of `[]u21`

lch361 commented 2025年07月07日 18:24:29 +02:00

Just a user of this library here. Amazing work!

In my opinion, ideally, uncompressed zg types probably could be made into global constants, because their content is always same. Having Graphemes, LetterCasing and friends as global variables would simplify library API a lot, lifting the burden of passing additional *const to every function.

It is now possible to define zg types in comptime, but for now it looks like this:

Just a user of this library here. Amazing work! In my opinion, ideally, uncompressed zg types probably could be made into global constants, because their content is always same. Having `Graphemes`, `LetterCasing` and friends as global variables would simplify library API a lot, lifting the burden of passing additional `*const` to every function. It is now possible to define zg types in comptime, but for now it looks like this: https://codeberg.org/lch361/spiral/src/commit/7507019b00e733e0c8a54ad07928717cfeaaefba/spiral-core/zg.zig#L4-L5

jacobsandlund commented 2025年07月08日 14:22:05 +02:00

@lch361 yes, I'm planning on PR'ing a change I made to not require these init's at all, since there is no allocation (aside from Normalize needing CanonData). But I'm wanting to wait until this PR and some of my others get merged in, since I didn't want to change the API in this PR that was already large.

You can see my full set of changes here, and note how now the API is just basic function calls: GeneralCategories.gc('A'). I could take it a step further and move all these modules into one zg one, having it then look something like zg.categories.gc('A') (renaming the GeneralCategories module to categories reflect it's not an instantiated struct), and let zig do the work to only keep what's used in the binary. That will be another PR, though.

@lch361 yes, I'm planning on PR'ing a change I made to not require these init's at all, since there is no allocation (aside from `Normalize` needing `CanonData`). But I'm wanting to wait until this PR and some of my others get merged in, since I didn't want to change the API in this PR that was already large. You can see my [full set of changes here](https://codeberg.org/atman/zg/compare/master...jacobsandlund:jacob), and note how now the API is just basic function calls: `GeneralCategories.gc('A')`. I could take it a step further and move all these modules into one `zg` one, having it then look something like `zg.categories.gc('A')` (renaming the `GeneralCategories` module to `categories` reflect it's not an instantiated struct), and let zig do the work to only keep what's used in the binary. That will be another PR, though.

atman commented 2025年07月08日 18:44:46 +02:00

I think this is overall a good idea, I'll leave this one open.

Did you benchmark the change to a two-tier lookup against the original? Time vs. space is usually a tradeoff, I suspect it is here.

Please (there's a theme you may be noticing) open an issue so we can discuss this one and track it. I've been leaning toward putting the zg data into static memory myself.

I think this is overall a good idea, I'll leave this one open. Did you benchmark the change to a two-tier lookup against the original? Time vs. space is usually a tradeoff, I suspect it is here. Please (there's a theme you may be noticing) open an issue so we can discuss this one and track it. I've been leaning toward putting the `zg` data into static memory myself.

jacobsandlund commented 2025年07月22日 04:57:28 +02:00

@atman, I've opened up #84 and #85 as two issues that this PR are addressing. Please especially see the former for a warm hello and apologies for jumping straight to opening PRs! Additionally I opened up #86 to discuss the API changes mentioned in this PR's description (but not yet PR'd as it depends on these changes first).

@atman, I've opened up https://codeberg.org/atman/zg/issues/84 and https://codeberg.org/atman/zg/issues/85 as two issues that this PR are addressing. Please especially see the former for a warm hello and apologies for jumping straight to opening PRs! Additionally I opened up https://codeberg.org/atman/zg/issues/86 to discuss the API changes mentioned in this PR's description (but not yet PR'd as it depends on these changes first).

atman commented 2025年12月23日 16:30:04 +01:00

@jacobsandlund could you get this PR current with 0.15.3? Ideally folded in with the other two issues, which it seems like you already have patches for. One PR is ideal. Thanks!

@jacobsandlund could you get this PR current with 0.15.3? Ideally folded in with the other two issues, which it seems like you already have patches for. One PR is ideal. Thanks!

jacobsandlund commented 2026年04月15日 16:13:08 +02:00

Hey @atman, sorry for the delay, and I'll drop the same message here as in the other PR: Feel free to take over this and do whatever you like with it!

Hey @atman, sorry for the delay, and I'll drop the same message here as in the [other PR](https://codeberg.org/atman/zg/pulls/75#issuecomment-13171806): Feel free to take over this and do whatever you like with it!

atman commented 2026年04月20日 22:18:47 +02:00

No worries @jacobsandlund! I wasn't able to merge the actual commits, too much Zig drift, but it's all in the next release, and I made it clear which contributions were yours in the NEWS file.

No worries @jacobsandlund! I wasn't able to merge the actual commits, too much Zig drift, but it's all in the next release, and I made it clear which contributions were yours in the NEWS file.

jacobsandlund commented 2026年04月21日 14:41:34 +02:00

Oh, that's great I'm glad you were able to pull it in @atman!

Oh, that's great I'm glad you were able to pull it in @atman!

atman commented 2026年04月21日 18:35:06 +02:00

Incorporated manually into v0.16.0. Thanks again Jacob.

Incorporated manually into [v0.16.0](https://codeberg.org/atman/zg/releases/tag/v0.16.0). Thanks again Jacob.

jacobsandlund commented 2026年04月23日 13:59:36 +02:00

oh wow just saw that NEWS.md, thank you for the shout-out @atman !

oh wow just saw that NEWS.md, thank you for the shout-out @atman !