MasonRemaley commented 2026年06月26日 00:18:04 +02:00

Intro

In #35914, I made some tweaks to std.meta.hasUniqueRepresentation(T: type) that got me thinking about this API.

This function returns true if it can be determined that T has a unique bit pattern for each value. There are two reasons I'm aware of that a type may not have a unique bit pattern for each value:

• It may contain padding. Since padding is undefined, the padding bits can be varied without varying the value.
• It may contain floats. NaN has multiple representations, as does zero.

Problem Statement

While the result of this function is comptime known, it is allowed to vary between builds, because it doesn't require that T have a well defined layout. For example, an untagged union might implicitly have a safety tag in a debug build that it loses in a release build, leading to padding in the debug build but not the release build.

This is a problem because it's easy to misuse in such a way as to write code that compiles in some optimization modes but not others.

Why not just require well defined layouts?

I initially thought that this was an oversight, and was going to add a check that T has a well defined layout to the function. However, this is actually at odds with about half of the usages in the compiler/standard library:

1. std.AutoHashMap and such accelerate hashing when a type has a unique representation by hashing the result of asBytes instead of iterating over the fields. Modifying this function to return false for non well defined layouts would likely pessimize our hash maps in release builds.
2. std.mem has a similar use case with similar tradeoffs for its searches.
3. std.Build.abi appears to use to use it to check that a type can be serialized. It first enforces that the type is extern compatible, which guarantees that it gets the same result for all optimization modes.
4. std.zig.Zoir has a similar use case with similar tradeoffs for its Header, Note, and CompileError type. It does not assert that these types are extern, but they are, so there's no issue here.
5. As of #35914, InternPool.zig uses this to assert that a simple hashing strategy is always correct for intern pool keys. Unlike hash map it has no fallback, but is still correct since it first asserts that the type is extern compatible.

Given that 50% of the cases are relying on this behavior it doesn't make sense to take it away. However, the other 50% of the cases are asserting that the layout is well defined as its required for those usages to be correct, and there's not a single catch all way to make this assert right now so they all use different strategies.

Additionally, in the use cases that don't involve hashing, there's no reason to disallow floats!

Proposal

This API is serving three distinct use cases right now. We should split it up such that each use case can be better served.

Here are the three use cases:

1. Deciding on a hashing strategy.
   • It's okay if the result varies between builds
   • Floats should not be allowed
2. Asserting that hash(asBytes(ptr)) strategy is viable with no fallback
   • Not okay to vary between builds
   • Floats should not be allowed
3. Asserting that serialized data has no padding
   • Not okay to vary between builds
   • Floats should be allowed

With these in mind, the following primitives would serve these use cases well:

• hasUniqueRepresentation -> std.mem.reprUnique (same semantics, new name)
• std.mem.reprDefined (new)
• std.mem.reprUnpadded (new)

Usage would be as follows:

• Callers choosing between hashing strategies would continue to use reprUnique (renamed from hasUniqueRepresentation)
• Callers asserting that a simple asBytes style hash is correct would use assert(reprDefined(T) and reprUnique(T))
• Callers checking their serialized types would use assert(reprDefined(T) and reprUnpadded(T))

Open Questions

Union vs Enum Tags

(削除) In #35914, @mlugg pointed out: (削除ここまで)

(削除) > [The] design of Zig does not require implementations to use the given tag type for the in-memory representation of a tagged union. For instance, a union(enum(u16)) with 256 or fewer fields could be represented in memory with a 1-byte tag---and this choice has no observable effects to the user, so is legal. (削除ここまで)

(削除) Is this also true for enums, or only for tagged unions? hasUniqueRepresentation currently assumes this is not true for enums. (削除ここまで)

(answered)

Explicit vs implicit tags

It's not currently possible to implement reprDefined in a useful way for packed structs, packed unions, or enums. These types are well defined if they have explicit tag types/backing ints, but std.lang.Type does not currently indicate whether the tag type/backing ints are explicit or implicit.

There are three ways we could resolve this. The first is to instead define this function as an assert. This works, but doesn't compose well with the other functions:

pubfnassertReprDefined(T:type)void{_=externstruct{well_defined_repr:T};};

The second is to modify std.lang.Type to include this information. This works, but we're ultimately just mirroring a bunch of logic that's already implemented in validateExtern in the compiler.

The third option is to expose a new building @reprDefined that shares code with Type.zig's validateExtern.

Future tooling to detect unsound memory operations

In #35914, @andrewrk pointed out:

I see, I suppose that is an OK definition of this function. This might bite us in the ass later though when we try to introduce tooling to find unsound operations on memory.

We can probably wait until we have such tools to resolve this, however, it's worth keeping this issue in mind because ideally we'd be able to introduce such tools without pessimizing hashing/searching, which might involve making tradeoffs about how exactly this check works.

Other Notes

• The inline TODO in hasUniqueRepresentation should be resolved
• The documentation on hasUniqueRepresentation should be improved to mention that the result can vary, be more explicit that floats are excluded, and should probably use the phrase "if it can be determine" since it is currently allowed to return false negatives.

# Intro In #35914, I made some tweaks to `std.meta.hasUniqueRepresentation(T: type)` that got me thinking about this API. This function returns true if it can be determined that `T` has a unique bit pattern for each value. There are two reasons I'm aware of that a type may not have a unique bit pattern for each value: * It may contain padding. Since padding is undefined, the padding bits can be varied without varying the value. * It may contain floats. `NaN` has multiple representations, as does zero. # Problem Statement While the result of this function is comptime known, it is allowed to vary between builds, because it doesn't require that `T` have a well defined layout. For example, an untagged union might implicitly have a safety tag in a debug build that it loses in a release build, leading to padding in the debug build but not the release build. This is a problem because it's easy to misuse in such a way as to write code that compiles in some optimization modes but not others. # Why not just require well defined layouts? I initially thought that this was an oversight, and was going to add a check that `T` has a well defined layout to the function. However, this is actually at odds with about half of the usages in the compiler/standard library: 1. `std.AutoHashMap` and such accelerate hashing when a type has a unique representation by hashing the result of `asBytes` instead of iterating over the fields. Modifying this function to return false for non well defined layouts would likely pessimize our hash maps in release builds. 2. `std.mem` has a similar use case with similar tradeoffs for its searches. 3. `std.Build.abi` appears to use to use it to check that a type can be serialized. It first enforces that the type is extern compatible, which guarantees that it gets the same result for all optimization modes. 4. `std.zig.Zoir` has a similar use case with similar tradeoffs for its `Header`, `Note`, and `CompileError` type. It does not assert that these types are extern, but they are, so there's no issue here. 5. As of #35914, `InternPool.zig` uses this to assert that a simple hashing strategy is always correct for intern pool keys. Unlike hash map it has no fallback, but is still correct since it first asserts that the type is extern compatible. Given that 50% of the cases are relying on this behavior it doesn't make sense to take it away. However, the other 50% of the cases are asserting that the layout *is* well defined as its required for those usages to be correct, and there's not a single catch all way to make this assert right now so they all use different strategies. **Additionally, in the use cases that don't involve hashing, there's no reason to disallow floats!** # Proposal This API is serving three distinct use cases right now. We should split it up such that each use case can be better served. Here are the three use cases: 1. Deciding on a hashing strategy. * It's **okay** if the result varies between builds * Floats **should not** be allowed 2. Asserting that `hash(asBytes(ptr))` strategy is viable with no fallback * **Not okay** to vary between builds * Floats **should not** be allowed 3. Asserting that serialized data has no padding * **Not okay** to vary between builds * Floats **should** be allowed With these in mind, the following primitives would serve these use cases well: * `hasUniqueRepresentation` -> `std.mem.reprUnique` (same semantics, new name) * `std.mem.reprDefined` (new) * `std.mem.reprUnpadded` (new) Usage would be as follows: * Callers choosing between hashing strategies would continue to use `reprUnique` (renamed from `hasUniqueRepresentation`) * Callers asserting that a simple `asBytes` style hash is correct would use `assert(reprDefined(T) and reprUnique(T))` * Callers checking their serialized types would use `assert(reprDefined(T) and reprUnpadded(T))` # Open Questions ### Union vs Enum Tags ~~In #35914, @mlugg [pointed out](https://codeberg.org/ziglang/zig/pulls/35914#issuecomment-17998925):~~ ~~> [The] design of Zig does not require implementations to use the given tag type for the in-memory representation of a tagged union. For instance, a union(enum(u16)) with 256 or fewer fields could be represented in memory with a 1-byte tag---and this choice has no observable effects to the user, so is legal.~~ ~~Is this also true for enums, or only for tagged unions? `hasUniqueRepresentation` currently assumes this is *not* true for enums.~~ [(answered)](https://codeberg.org/ziglang/zig/issues/35944#issuecomment-18091973) ### Explicit vs implicit tags It's not currently possible to implement `reprDefined` in a useful way for packed structs, packed unions, or enums. These types are well defined if they have explicit tag types/backing ints, but `std.lang.Type` does not currently indicate whether the tag type/backing ints are explicit or implicit. There are three ways we could resolve this. The first is to instead define this function as an assert. This works, but doesn't compose well with the other functions: ```zig pub fn assertReprDefined(T: type) void { _ = extern struct { well_defined_repr: T }; }; ``` The second is to modify `std.lang.Type` to include this information. This works, but we're ultimately just mirroring a bunch of logic that's already implemented in `validateExtern` in the compiler. The third option is to expose a new building `@reprDefined` that shares code with `Type.zig`'s `validateExtern`. ### Future tooling to detect unsound memory operations In #35914, @andrewrk [pointed out](https://codeberg.org/ziglang/zig/pulls/35914#issuecomment-18072224): > I see, I suppose that is an OK definition of this function. This might bite us in the ass later though when we try to introduce tooling to find unsound operations on memory. We can probably wait until we have such tools to resolve this, however, it's worth keeping this issue in mind because ideally we'd be able to introduce such tools without pessimizing hashing/searching, which might involve making tradeoffs about how exactly this check works. # Other Notes * [ ] The inline TODO in `hasUniqueRepresentation` should be resolved * [ ] The documentation on `hasUniqueRepresentation` should be improved to mention that the result can vary, be more explicit that floats are excluded, and should probably use the phrase "if it can be determine" since it is currently allowed to return false negatives.

andrewrk commented 2026年06月26日 00:40:18 +02:00

By the way we started using "proposal" only for language changes. I understand you're still looking for discussion here, but that's kind of already implicitly the case on all the issues until they get sorted; there isn't really a label for planned vs unplanned enhancements.

Your suggestion sounds like an improvement; even if we don't end up doing exactly what you said, it's the right next step to take, I think.

I suggest putting the functions in std.mem instead of std.meta and name them:

• std.mem.reprUnique
• std.mem.reprDefined
• std.mem.reprUnpadded (note that "zero padding" is ambiguous and therefore that phrasing should not be used)

This follows my favorite naming convention: related functionality shares a common prefix, each distinction is represented by exactly one word, and grammatical constructs are absent. Thanks to std.mem namespace, the meaning is concise yet unambiguous:

• "memory representation unique"
• "memory representation defined"
• "memory representation unpadded"

By the way we started using "proposal" only for language changes. I understand you're still looking for discussion here, but that's kind of already implicitly the case on all the issues until they get sorted; there isn't really a label for planned vs unplanned enhancements. Your suggestion sounds like an improvement; even if we don't end up doing exactly what you said, it's the right next step to take, I think. I suggest putting the functions in `std.mem` instead of `std.meta` and name them: * `std.mem.reprUnique` * `std.mem.reprDefined` * `std.mem.reprUnpadded` (note that "zero padding" is ambiguous and therefore that phrasing should not be used) This follows my favorite naming convention: related functionality shares a common prefix, each distinction is represented by exactly one word, and grammatical constructs are absent. Thanks to `std.mem` namespace, the meaning is concise yet unambiguous: * "memory representation unique" * "memory representation defined" * "memory representation unpadded"

mlugg commented 2026年06月26日 08:13:48 +02:00

In #35914, @mlugg pointed out:

[The] design of Zig does not require implementations to use the given tag type for the in-memory representation of a tagged union. For instance, a union(enum(u16)) with 256 or fewer fields could be represented in memory with a 1-byte tag---and this choice has no observable effects to the user, so is legal.

Is this also true for enums, or only for tagged unions? hasUniqueRepresentation currently assumes this is not true for enums.

Only tagged unions. It's a core guarantee of the language that enum(T) has the same bit-level and byte-level representation as T.

The union(enum(T)) quirk comes about as a result of the following properties:

• union(enum(T)) does not have a well-defined layout, so it is incorrect to assume that the tag can be found at any particular offset.
• Given a pointer to a union(enum(T)), there is no way to construct a pointer to its enum(T) tag in memory.

These mean that there is no legal way to actually get a pointer to the tag of a tagged union in memory, which, as a natural consequence, frees up implementations to represent them however they want. By the way, the same is true for the error set inside of error unions---the memory layout of the type E!T doesn't technically need to contain an E verbatim in its memory layout, because there is no legal way to construct a pointer to that part of an error union!

> In #35914, @mlugg [pointed out](https://codeberg.org/ziglang/zig/pulls/35914#issuecomment-17998925): > > > [The] design of Zig does not require implementations to use the given tag type for the in-memory representation of a tagged union. For instance, a union(enum(u16)) with 256 or fewer fields could be represented in memory with a 1-byte tag---and this choice has no observable effects to the user, so is legal. > > Is this also true for enums, or only for tagged unions? hasUniqueRepresentation currently assumes this is not true for enums. Only tagged unions. It's a core guarantee of the language that `enum(T)` has the same bit-level and byte-level representation as `T`. The `union(enum(T))` quirk comes about as a result of the following properties: * `union(enum(T))` does not have a well-defined layout, so it is incorrect to assume that the tag can be found at any particular offset. * Given a pointer to a `union(enum(T))`, there is no way to construct a pointer to its `enum(T)` tag in memory. These mean that there is no legal way to actually get a pointer to the tag of a tagged union in memory, which, as a natural consequence, frees up implementations to represent them however they want. By the way, the same is true for the error set inside of error unions---the memory layout of the type `E!T` doesn't technically need to contain an `E` verbatim in its memory layout, because there is no legal way to construct a pointer to that part of an error union!

Fri3dNstuff commented 2026年06月26日 11:43:06 +02:00

I am very happy to see that the use cases for the current behaviour of hasUniqueRepresentation are acknowledged - being able to opportunistically reinterpret types (even ones which are not allowed in extern positions) as u8 arrays is very useful for (as described in the issue's statement) accelerating hashing and equality, as well as potentially avoiding some monomorphisation bloat in hash-map implementations.

I have two questions/suggestions, one about the behaviour of floats, and the other about the meaning of "padding bits".

Regarding floats, this is perhaps a wild proposal, so feel free to shut this down immediately. Can we consider redefining the meaning of the operators ==, !=, >, >=, <, and <= for floating pointer operands, and vectors thereof? As I understand, currently Zig bases the operators' behaviour on IEEE-754 2019 §5.11: Comparison Predicates. The big issue with that definition, in my opinion, is that the == operator does not act like equality: it is not reflexive, as NaN != NaN, even if both operands are bitwise identical; it also does not follow the usual rule that x = y → f(x) = f(y), because 0.0 and -0.0 are defined to be equal, and yet 1.0 / 0.0 == ∞, 1.0 / -0.0 == -∞, ∞ != -∞. These confusing semantics cause bugs, bugs currently found in Zig's standard library: consider std.mem.eql, which short-circuits when its two inputs are slices to the same memory. This optimisation, which is incorrect when working on floats, causes this assertion to fail:

constx:[1]f64=.{std.math.nan(f64)};std.debug.assert(!std.mem.eql(f64,&x,&x));

IEEE-754 provides a definition for a "totalOrder" predicate (at §5.10), which essentially treats the given operands' datum as sign-and-magnitude integers, with the additional detail that 0.0 > -0.0 - some wiggle room is given for NaNs. Importantly, NaNs of identical bit-layout are judged equal, and as said, 0.0 > -0.0. Redefining Zig's operators to be based on totalOrder solves the bug in std.mem.eql, and importantly for this issue: it makes hasWellDefinedRepresentation and hasZeroPadding's behaviours identical, allowing us to merge the two.

Regarding padding bits, I am yet to come across a comprehensive explanation of what those even are! I'm concerned that people might be taking past each other when using the term - this might lead to some misunderstandings, which can cause subtle compiler bugs, or conflicting language proposals. Padding bytes are a well-known concept, found in C, C++, and Rust - following the Rust model, they are used (among other things) to align struct fields to their desired alignment, they are uninitialised, and writes of said struct types reset those paddings to be uninitialised, in cases those were initialised directly by other means. Continuing with Rust (just because I find their formalisation efforts to be a lot more serious than the prose-filled specification of C), memory initialisation has byte-level granularity - a consequence is that booleans in Rust have a unique representation. This is in contrast to hasUniqueRepresentation's current answer (false) for a bool input. My question, then, is: how is the Abstract Byte defined in Zig's Abstract Execution Machine? An answer to that question is required in order to determine hasUniqueRepresentation's result for many input types (including booleans, exotic integers, some enums, packed structs, packed unions, and types constructed therefrom).

I am very happy to see that the use cases for the current behaviour of `hasUniqueRepresentation` are acknowledged - being able to opportunistically reinterpret types (even ones which are not allowed in `extern` positions) as `u8` arrays is very useful for (as described in the issue's statement) accelerating hashing and equality, as well as potentially avoiding some monomorphisation bloat in hash-map implementations. I have two questions/suggestions, one about the behaviour of floats, and the other about the meaning of "padding bits". Regarding floats, this is perhaps a wild proposal, so feel free to shut this down immediately. Can we consider redefining the meaning of the operators `==`, `!=`, `>`, `>=`, `<`, and `<=` for floating pointer operands, and vectors thereof? As I understand, currently Zig bases the operators' behaviour on [IEEE-754 2019](https://ieeexplore.ieee.org/document/8766229) §5.11: Comparison Predicates. The big issue with that definition, in my opinion, is that the `==` operator does not act like equality: it is not [reflexive](https://en.wikipedia.org/wiki/Reflexive_relation), as `NaN != NaN`, even if both operands are bitwise identical; it also does not follow the usual rule that `x = y → f(x) = f(y)`, because `0.0` and `-0.0` are defined to be equal, and yet `1.0 / 0.0 == ∞`, `1.0 / -0.0 == -∞`, `∞ != -∞`. *These confusing semantics cause bugs*, bugs currently found in Zig's standard library: consider [`std.mem.eql`](https://ziglang.org/documentation/master/std/#std.mem.eql), which short-circuits when its two inputs are slices to the same memory. This optimisation, which is incorrect when working on floats, causes this assertion to fail: ```zig const x: [1]f64 = .{ std.math.nan(f64) }; std.debug.assert(!std.mem.eql(f64, &x, &x)); ``` IEEE-754 provides a definition for a "`totalOrder`" predicate (at §5.10), which essentially treats the given operands' datum as sign-and-magnitude integers, with the additional detail that `0.0 > -0.0` - some wiggle room is given for `NaN`s. Importantly, `NaN`s of identical bit-layout are judged equal, and as said, `0.0 > -0.0`. Redefining Zig's operators to be based on `totalOrder` solves the bug in `std.mem.eql`, and importantly for this issue: *it makes `hasWellDefinedRepresentation` and `hasZeroPadding`'s behaviours identical*, allowing us to merge the two. Regarding padding bits, I am yet to come across a comprehensive explanation of what those even are! I'm concerned that people might be taking past each other when using the term - this might lead to some misunderstandings, which can cause subtle compiler bugs, or conflicting language proposals. Padding *bytes* are a well-known concept, found in C, C++, and Rust - following the Rust model, they are used (among other things) to align `struct` fields to their desired alignment, they are [uninitialised](https://www.ralfj.de/blog/2019/07/14/uninit.html), and writes of said `struct` types reset those paddings to be uninitialised, in cases those were initialised directly by other means. Continuing with Rust (just because I find their formalisation efforts to be a lot more serious than the prose-filled specification of C), [memory initialisation has byte-level granularity](https://github.com/minirust/minirust/blob/master/spec/mem/interface.md#abstract-bytes) - a consequence is that `bool`eans in Rust have a unique representation. This is in contrast to `hasUniqueRepresentation`'s current answer (`false`) for a `bool` input. My question, then, is: *how is the Abstract Byte defined in Zig's Abstract Execution Machine?* An answer to that question is required in order to determine `hasUniqueRepresentation`'s result for many input types (including `bool`eans, exotic integers, some `enum`s, `packed struct`s, `packed union`s, and types constructed therefrom).

MasonRemaley commented 2026年06月26日 13:06:47 +02:00

@andrewrk wrote in #35944 (comment):

I suggest putting the functions in std.mem instead of std.meta and name them:
[...]

I like these names/namespacing. Less typing, much clearer, and good point about the term "hasZeroPadding" being ambiguous. I've edited the original post to use them.

---

@mlugg wrote in #35944 (comment):

Only tagged unions. It's a core guarantee of the language that enum(T) has the same bit-level and byte-level representation as T. [..]

That makes sense, thanks for the explanation!

---

@Fri3dNstuff wrote in #35944 (comment):

Regarding floats, this is perhaps a wild proposal, so feel free to shut this down immediately. [...]

I don't see a future where these operators are redefined, but I'm sympathetic to the idea of providing a more advanced std.meta.eql that takes an options struct or enum. Details are out of scope for this thread though.

---

@Fri3dNstuff wrote in #35944 (comment):

Regarding padding bits, I am yet to come across a comprehensive explanation of what those even are! [...]

Good point, we should define our terms here as this may impact how these methods are defined. When I use the term "padding bits", I'm referring to bits that are counted by @sizeOf(T) but left undefined.

Here are some examples:

• A u8 has no padding bits, @sizeOf(u8) == 1 and the full byte is defined.
• A f32 has no padding bits, @sizeOf(f32) == 4 and the full 4 bytes are defined. (The fact that f32 doesn't have a unique representation isn't relevant.)
• If @sizeOf(u7) == 1, then u7 has exactly 1 padding bit that is left undefined.
• A bool has no padding bits. @sizeOf(bool) == 1, and while it is true that you could store a bool in one bit, writing to one writes all 8 bits.
• A extern struct { a: u8, b: u16 } typically has 8 bits of padding following a to keep b: u16 aligned, these extra 8 bits are undefined.

With this in mind, reprUnique (formerly hasUniqueRepresention) should be changed to return true for bool. There are only two valid bit patterns for bool, true == 0b0000_0001 and false == 0b0000_0000. I chatted with @mlugg about this offline--I don't think these semantics are explicitly specified anywhere right now, but they seem correct and empirically are how the compiler currently behaves.

---

A tradeoff from the original post that I'm particularly interested in feedback on is std.mem.reprDefined vs @reprDefined. See Explicit vs implicit tags header in the OP, I buried it a little but the key point is that it's not actually possible to implement reprDefined in user space today so we either need to modify the language to make this possible or we need to supply it as a builtin.

@andrewrk wrote in https://codeberg.org/ziglang/zig/issues/35944#issuecomment-18082523: > I suggest putting the functions in `std.mem` instead of `std.meta` and name them: > [...] I like these names/namespacing. Less typing, much clearer, and good point about the term "hasZeroPadding" being ambiguous. I've edited the original post to use them. *** @mlugg wrote in https://codeberg.org/ziglang/zig/issues/35944#issuecomment-18091973: > Only tagged unions. It's a core guarantee of the language that `enum(T)` has the same bit-level and byte-level representation as `T`. [..] That makes sense, thanks for the explanation! *** @Fri3dNstuff wrote in https://codeberg.org/ziglang/zig/issues/35944#issuecomment-18099263: > Regarding floats, this is perhaps a wild proposal, so feel free to shut this down immediately. [...] I don't see a future where these operators are redefined, but I'm sympathetic to the idea of providing a more advanced `std.meta.eql` that takes an options struct or enum. Details are out of scope for this thread though. *** @Fri3dNstuff wrote in https://codeberg.org/ziglang/zig/issues/35944#issuecomment-18099263: > Regarding padding bits, I am yet to come across a comprehensive explanation of what those even are! [...] Good point, we should define our terms here as this may impact how these methods are defined. When I use the term "padding bits", I'm referring to bits that are counted by `@sizeOf(T)` but left undefined. Here are some examples: * A `u8` has no padding bits, `@sizeOf(u8) == 1` and the full byte is defined. * A `f32` has no padding bits, `@sizeOf(f32) == 4` and the full 4 bytes are defined. (The fact that `f32` doesn't have a unique representation isn't relevant.) * If `@sizeOf(u7) == 1`, then `u7` has exactly 1 padding bit that is left undefined. * A `bool` has no padding bits. `@sizeOf(bool) == 1`, and while it is true that you *could* store a bool in one bit, writing to one writes all 8 bits. * A `extern struct { a: u8, b: u16 }` typically has 8 bits of padding following `a` to keep `b: u16` aligned, these extra 8 bits are undefined. With this in mind, `reprUnique` (formerly `hasUniqueRepresention`) should be changed to return `true` for `bool`. There are only two valid bit patterns for `bool`, `true == 0b0000_0001` and `false == 0b0000_0000`. I chatted with @mlugg about this offline--I don't think these semantics are explicitly specified anywhere right now, but they seem correct and empirically are how the compiler currently behaves. *** A tradeoff from the original post that I'm particularly interested in feedback on is `std.mem.reprDefined` vs `@reprDefined`. See `Explicit vs implicit tags` header in the OP, I buried it a little but the key point is that it's not actually possible to implement `reprDefined` in user space today so we either need to modify the language to make this possible or we need to supply it as a builtin.

justusk commented 2026年06月26日 15:26:27 +02:00

A tradeoff from the original post that I'm particularly interested in feedback on is std.mem.reprDefined vs @reprDefined.
[...] we either need to modify the language to make this possible or we need to supply it as a builtin.

I am in favor of modifying the language. std.lang.Type should IMO just set the backing integer type to null for implicit backing integers (implying that std.lang.Type.Enum.tag_type should become an optional).

The compiler obviously has to choose a backing integer with a fitting bit count for the implicit case at some point, but I see no real reason to expose that choice to the user. Whether the backing integer chosen by the compiler is signed or unsigned shouldn't be of concern to the user because types with implicit backing integers are not allowed in an extern context and are generally not considered to have a defined bit-level representation anyway. The implicit backing integer would still be observable via @intFromEnum, but if #35602 is implemented as described here that won't be the case for its successor @backingInt anymore.

Also a user-space implementation of reprDefined has the advantage that its exact rules are more observable to most Zig users as there are going to be more people who are able/willing to read std code than compiler source code.

> A tradeoff from the original post that I'm particularly interested in feedback on is `std.mem.reprDefined` vs `@reprDefined`. > [...] we either need to modify the language to make this possible or we need to supply it as a builtin. I am in favor of modifying the language. `std.lang.Type` should IMO just set the backing integer type to `null` for implicit backing integers (implying that `std.lang.Type.Enum.tag_type` should become an optional). The compiler obviously has to choose a backing integer with a fitting bit count for the implicit case at some point, but I see no real reason to expose that choice to the user. Whether the backing integer chosen by the compiler is signed or unsigned shouldn't be of concern to the user because types with implicit backing integers are not allowed in an `extern` context and are generally not considered to have a defined bit-level representation anyway. The implicit backing integer would still be observable via `@intFromEnum`, but if #35602 is implemented as described [here](https://codeberg.org/ziglang/zig/issues/35602#issuecomment-16530587) that won't be the case for its successor `@backingInt` anymore. Also a user-space implementation of `reprDefined` has the advantage that its exact rules are more observable to most Zig users as there are going to be more people who are able/willing to read std code than compiler source code.

Fri3dNstuff commented 2026年06月26日 15:39:24 +02:00

@MasonRemaley

I don't see a future where these operators are redefined, [...]

Very understandable - I've opened a PR, then, to fix std.mem.eql.

If @sizeOf(u7) == 1, then u7 has exactly 1 padding bit that is left undefined.

That means that Zig sees undefined at a bit-level granularity, which is unlike C and Rust. This also means that std.mem.swap is buggy: the function internally interprets its two arguments as (aligned) []u8s, then swaps the slices' contents one u8 pair at a time. Assuming that a u8 value (a value - importantly, not a u8 living in the Abstract Machine's main memory!) can only hold one of 257 values (0 through 255, as well as undefined), and that loading an integer with any undefined bits causes the whole int to become undefined, a swap of two u7s would lead to them both becoming undefined:

vara:u7=1;varb:u7=2;std.mem.swap(u7,&a,&b);// both `a` and `b` are now `undefined`if(a==2)std.debug.print("sneaky IB!\n",.{});

@MasonRemaley > I don't see a future where these operators are redefined, [...] Very understandable - I've opened [a PR](https://codeberg.org/ziglang/zig/pulls/35947), then, to fix `std.mem.eql`. > If `@sizeOf(u7) == 1`, then `u7` has exactly 1 padding bit that is left undefined. That means that Zig sees `undefined` at a bit-level granularity, which is unlike C and Rust. This also means that [`std.mem.swap`](https://ziglang.org/documentation/master/std/#std.mem.swap) is buggy: the function internally interprets its two arguments as (aligned) `[]u8`s, then swaps the slices' contents one `u8` pair at a time. Assuming that a `u8` value (a value - importantly, not a `u8` living in the Abstract Machine's main memory!) can only hold one of 257 values (`0` through `255`, as well as `undefined`), and that loading an integer with any `undefined` bits [causes the whole int to become `undefined`](https://github.com/ziglang/zig/issues/19634), a swap of two `u7`s would lead to them both becoming `undefined`: ```zig var a: u7 = 1; var b: u7 = 2; std.mem.swap(u7, &a, &b); // both `a` and `b` are now `undefined` if (a == 2) std.debug.print("sneaky IB!\n", .{}); ```

MasonRemaley commented 2026年06月26日 22:58:51 +02:00

@Fri3dNstuff wrote in #35944 (comment):

If @sizeOf(u7) == 1, then u7 has exactly 1 padding bit that is left undefined.

That means that Zig sees undefined at a bit-level granularity, which is unlike C and Rust. [...]

I think this problem is worth discussing further in github/#19634, but I don't think it's a blocker for this issue. Whatever resolution we end up with will allow for loading and storing all supported integer types, but will not define the value of the unused bits, so the implementations of reprUnique/reprDefined/reprUnpadded should remain unaffected.

(It's possible I'm missing something though, let me know if I've misunderstood the issue.)

---

@justusk wrote in #35944 (comment):

The compiler obviously has to choose a backing integer with a fitting bit count for the implicit case at some point, but I see no real reason to expose that choice to the user. Whether the backing integer chosen by the compiler is signed or unsigned shouldn't be of concern to the user because types with implicit backing integers are not allowed in an extern context and are generally not considered to have a defined bit-level representation anyway. The implicit backing integer would still be observable via @intFromEnum, but if #35602 is implemented as described here that won't be the case for its successor @backingInt anymore.

This argument makes sense to me. If the bit-level representation is not defined for packed unions with implicit backing ints, then I see no reason to expose the backing int type to the user when it's implicit. They can still check whether the union is packed or not using the layout field.

Unlike packed unions, I think we do need to add information to the type info for enums if we want reprDefined to be implementable in user space. The tag type for an enum is useful information whether it's implicit or explicit, so we'd likely want to introduce a tag_explicit: bool field or such.

@Fri3dNstuff wrote in https://codeberg.org/ziglang/zig/issues/35944#issuecomment-18108020: > > If `@sizeOf(u7) == 1`, then `u7` has exactly 1 padding bit that is left undefined. > > That means that Zig sees `undefined` at a bit-level granularity, which is unlike C and Rust. [...] I think this problem is worth discussing further in [github/#19634](https://github.com/ziglang/zig/issues/19634), but I don't think it's a blocker for this issue. Whatever resolution we end up with will allow for loading and storing all supported integer types, but will not define the value of the unused bits, so the implementations of `reprUnique`/`reprDefined`/`reprUnpadded` should remain unaffected. (It's possible I'm missing something though, let me know if I've misunderstood the issue.) *** @justusk wrote in https://codeberg.org/ziglang/zig/issues/35944#issuecomment-18107558: > The compiler obviously has to choose a backing integer with a fitting bit count for the implicit case at some point, but I see no real reason to expose that choice to the user. Whether the backing integer chosen by the compiler is signed or unsigned shouldn't be of concern to the user because types with implicit backing integers are not allowed in an `extern` context and are generally not considered to have a defined bit-level representation anyway. The implicit backing integer would still be observable via `@intFromEnum`, but if #35602 is implemented as described [here](https://codeberg.org/ziglang/zig/issues/35602#issuecomment-16530587) that won't be the case for its successor `@backingInt` anymore. This argument makes sense to me. If the bit-level representation is not defined for packed unions with implicit backing ints, then I see no reason to expose the backing int type to the user when it's implicit. They can still check whether the union is packed or not using the layout field. Unlike packed unions, I think we do need to add information to the type info for enums if we want `reprDefined` to be implementable in user space. The tag type for an enum is useful information whether it's implicit or explicit, so we'd likely want to introduce a `tag_explicit: bool` field or such.

mlugg commented 2026年06月27日 09:38:36 +02:00

On the bit-level-undefined stuff: none of this is 100% settled yet, but to my knowledge, we're trying to avoid having bit-level definedness in Zig---in part because we need to track undefined values at comptime, and having partially-undefined values really blows up the implementation complexity of the compiler. What we've been calling "padding bits" (which I would say is not a particularly precise term) have unspecified values---implementations may, if they wish, depend on them having specific values. This is very important for efficiency, and in fact is how the Zig compiler works today---right now, @sizeOf(u17) == 4, and those 4 bytes actually contain a 32-bit integer where the high 15 bits must all be 0 for a well-defined u17 value.

You might then ask what happens if those bits aren't all zero when you dereference a pointer. My take on the semantics, which other core team members may not agree with (so don't take this as confirmation of the semantics!), is that when loading a value from memory, if the loaded bytes do not correspond to a valid value for the loaded type, then the value loaded is considered to be equivalent to undefined. For instance:

• bool values occupy 1 byte in memory, and have two valid bit patterns: 0b0000_0000 for false, and 0b0000_0001 for true. Dereferencing a *bool where the pointed-to byte is e.g. 0b0000_0010 results in @as(bool, undefined).
• The layout of u17 is implementation-specified, but in our compiler implementation it occupies 4 bytes in memory, according to the zero-extension rule I mentioned above. Dereferencing a *u17 where one of the 15 most-significant bits (in native endian) is non-zero results in the value @as(u17, undefined).

On the bit-level-`undefined` stuff: none of this is 100% settled yet, but to my knowledge, we're trying to avoid having bit-level definedness in Zig---in part because we need to track `undefined` values at `comptime`, and having partially-`undefined` values really blows up the implementation complexity of the compiler. What we've been calling "padding bits" (which I would say is not a particularly precise term) have *unspecified* values---implementations may, if they wish, depend on them having specific values. This is very important for efficiency, and in fact is how the Zig compiler works today---right now, `@sizeOf(u17) == 4`, and those 4 bytes actually contain a 32-bit integer where the high 15 bits *must* all be 0 for a well-defined `u17` value. You might then ask what happens if those bits *aren't* all zero when you dereference a pointer. **My take on the semantics, which other core team members may not agree with** (so don't take this as confirmation of the semantics!), is that when loading a value from memory, if the loaded bytes do not correspond to a valid value for the loaded type, then the value loaded is considered to be equivalent to `undefined`. For instance: * `bool` values occupy 1 byte in memory, and have two valid bit patterns: `0b0000_0000` for `false`, and `0b0000_0001` for `true`. Dereferencing a `*bool` where the pointed-to byte is e.g. `0b0000_0010` results in `@as(bool, undefined)`. * The layout of `u17` is implementation-specified, but in our compiler implementation it occupies 4 bytes in memory, according to the zero-extension rule I mentioned above. Dereferencing a `*u17` where one of the 15 most-significant bits (in native endian) is non-zero results in the value `@as(u17, undefined)`.

mlugg commented 2026年06月27日 09:55:28 +02:00

types with implicit backing integers [...] are generally not considered to have a defined bit-level representation anyway

That's not quite true, but I understand the confusion, because these definitions have never really been spelled out (and also the compiler source code is, um, wrong about some of the details at the moment). For the avoidance of doubt, here's what I believe are the current language rules in this area. Again, it's possible there's disagreement within the core team about some of this, but this is my interpretation of current language rules and from reading & writing & implementing several proposals in this area.

(EDIT: oh, I should also mention that this doesn't affect the actual points you were making---I agree that we should not expose a generated backing integer type for packed struct/packed union, and I think the points being made are fine if you replace "bit-level representation" with "in-memory representation" or something like that)

Types in Zig may have a "byte representation"/"memory representation" and a "bit representation". The former specifies how a value of that type is represented as a sequence of @sizeOf(T) bytes in memory, while the latter specifies how a value of that type is handled in bitpack types and by @bitCast.

Every type which is not comptime-only has a memory representation, but many such types have "ill-defined" memory representation (in contrast to "well-defined"). This means that the bytes used are not specified by the Zig language specification, and are considered to be up to the implementation. u17 is an example of such a type.

Many types do not have a bit representation. The only types which do have one are, basically, things you can @bitCast.

Here's a table classifying things:

memory representation?	bit representation?
comptime_int	no	no
u8	yes, well-defined	yes (LSB first)
u17	yes, ill-defined	yes (LSB first)
packed struct(T)	yes, same as T	yes (fields in order; or equivalently, same as T)
packed union(T)	yes, same as T	yes (fields in order; or equivalently, same as T)
packed struct	yes, ill-defined	yes (fields in order)
packed union	yes, ill-defined	yes (fields in order)
enum(T)	yes, same as T	yes (same as T)
enum	yes, ill-defined	no

Note that packed struct with unspecified backing type has ill-defined memory representation, but it still has a well-defined bit representation---this is very important, because the type would be next to useless otherwise (e.g. you wouldn't be able to @bitCast it, or to embed it in another bitpack type).

> types with implicit backing integers [...] are generally not considered to have a defined bit-level representation anyway That's not quite true, but I understand the confusion, because these definitions have never really been spelled out (and also the compiler source code is, um, wrong about some of the details at the moment). For the avoidance of doubt, here's what I believe are the current language rules in this area. **Again, it's possible there's disagreement within the core team about some of this, but this is my interpretation of current language rules and from reading & writing & implementing several proposals in this area.** (EDIT: oh, I should also mention that this doesn't affect the actual points you were making---I agree that we should not expose a generated backing integer type for `packed struct`/`packed union`, and I think the points being made are fine if you replace "bit-level representation" with "in-memory representation" or something like that) Types in Zig may have a "byte representation"/"memory representation" and a "bit representation". The former specifies how a value of that type is represented as a sequence of `@sizeOf(T)` bytes in memory, while the latter specifies how a value of that type is handled in `bitpack` types and by `@bitCast`. Every type which is not comptime-only has a memory representation, but many such types have "ill-defined" memory representation (in contrast to "well-defined"). This means that the bytes used are not specified by the Zig language specification, and are considered to be up to the implementation. `u17` is an example of such a type. Many types do *not* have a bit representation. The only types which do have one are, basically, things you can `@bitCast`. Here's a table classifying things: | | memory representation? | bit representation? | |---|----------|-------| | `comptime_int` | no | no | | `u8` | yes, well-defined | yes (LSB first) | | `u17` | yes, ill-defined | yes (LSB first) | | `packed struct(T)` | yes, same as T | yes (fields in order; or equivalently, same as T) | | `packed union(T)` | yes, same as T | yes (fields in order; or equivalently, same as T) | | `packed struct` | yes, ill-defined | yes (fields in order) | | `packed union` | yes, ill-defined | yes (fields in order) | | `enum(T)` | yes, same as T | yes (same as T) | | `enum` | yes, ill-defined | no | Note that `packed struct` with unspecified backing type has ill-defined memory representation, but it still has a well-defined bit representation---this is very important, because the type would be next to useless otherwise (e.g. you wouldn't be able to `@bitCast` it, or to embed it in another bitpack type).