There are too many integer parsing implementations in the standard library #30881

rpkak commented 2026年01月18日 12:34:32 +01:00

#12446
#24687

In addition to std.fmt.parse* functions there are in no particular order:

• std.crypto.Certificate.parseTimeDigits
• std.crypto.Certificate.parseYear4
• std.http.Client.Response.Head.parseInt3
• std.Io.net.Ip4Address.parse
• std.Io.net.Ip6Address.parse
• std.fmt.Parser.number
• here, here, here, here, here, and here in std.fmt.parse_float
• std.http.ChunkParser.feed
• std.zig.number_literal.parseNumberLiteral
• Parsing of escape sequences in strings in zig and json
• std.zig.AstGen.parseBitCount
• std.process: parsing uid and gid from /etc/passwd
• one implementation used by libzigc for ato* and strto* functions (#30792).
• std.math.big.int.Mutable.setString
• #30791
• more in aro, translate-c and resinator
• maybe more that I haven't found

One problem with such code duplication is that more code means more bugs. While writing this, I found #30880.

Some of the implementations above (like std.math.big.int.Mutable.setString) are likely too specific to benefit from a common implementation, but many of these use cases could use a common implementation.

Some properties that, in my opinion, should be configurable in a common implementation (feel free to argue against these or in favour of other properties):

• Should symbols like _ be allowed between digits? Which symbols (', _)?
• Are + and - allowed?
• What base is the number? Should prefixes like 0x, 0o and 0b change the base to 16, 8 and 2.
• Is a leading 0 allowed? Should it change the base to 8?
• Is the complete input string a number or only the beginning of the input string?

Like the author of #24687, I'm not sure what the ideal API would look like. #12446 proposed a config struct.

There are both use cases, for the base to be comptime known and for the base to be known only at runtime (and for the base to be specified by a prefix).

For some use cases the exact length of the input string is comptime known, for other use cases it is not.

For some use cases like std.process.posixGetUserInfoPasswdStream, it might make sense to use a std.Io.Reader instead of a string as the input, but in other use cases, parsing from a fixed reader would probably generate worse machine code than parsing from a string. Also, since string representations of integers are not very long in most use cases, it might make sense in general to first create a []const u8 of the input.

[#12446](https://github.com/ziglang/zig/issues/12446) [#24687](https://github.com/ziglang/zig/issues/24687) In addition to `std.fmt.parse*` functions there are in no particular order: - [`std.crypto.Certificate.parseTimeDigits`](https://codeberg.org/ziglang/zig/src/commit/e5dc5a6eb5608c100e78d6116942a4cd17f56d00/lib/std/crypto/Certificate.zig#L653-L667) - [`std.crypto.Certificate.parseYear4`](https://codeberg.org/ziglang/zig/src/commit/e5dc5a6eb5608c100e78d6116942a4cd17f56d00/lib/std/crypto/Certificate.zig#L682-L694) - [`std.http.Client.Response.Head.parseInt3`](https://codeberg.org/ziglang/zig/src/commit/e5dc5a6eb5608c100e78d6116942a4cd17f56d00/lib/std/http/Client.zig#L694-L699) - [`std.Io.net.Ip4Address.parse`](https://codeberg.org/ziglang/zig/src/commit/e5dc5a6eb5608c100e78d6116942a4cd17f56d00/lib/std/Io/net.zig#L354-L384) - [`std.Io.net.Ip6Address.parse`](https://codeberg.org/ziglang/zig/src/commit/e5dc5a6eb5608c100e78d6116942a4cd17f56d00/lib/std/Io/net.zig#L465-L572) - [`std.fmt.Parser.number`](https://codeberg.org/ziglang/zig/src/commit/e5dc5a6eb5608c100e78d6116942a4cd17f56d00/lib/std/fmt.zig#L167-L181) - [here](https://codeberg.org/ziglang/zig/src/commit/e5dc5a6eb5608c100e78d6116942a4cd17f56d00/lib/std/fmt/parse_float/parse.zig#L43-L46), [here](https://codeberg.org/ziglang/zig/src/commit/e5dc5a6eb5608c100e78d6116942a4cd17f56d00/lib/std/fmt/parse_float/parse.zig#L58-L65), [here](https://codeberg.org/ziglang/zig/src/commit/e5dc5a6eb5608c100e78d6116942a4cd17f56d00/lib/std/fmt/parse_float/parse.zig#L80-L85), [here](https://codeberg.org/ziglang/zig/src/commit/e5dc5a6eb5608c100e78d6116942a4cd17f56d00/lib/std/fmt/parse_float/decimal.zig#L245-L247), [here](https://codeberg.org/ziglang/zig/src/commit/e5dc5a6eb5608c100e78d6116942a4cd17f56d00/lib/std/fmt/parse_float/decimal.zig#L268-L270), and [here](https://codeberg.org/ziglang/zig/src/commit/e5dc5a6eb5608c100e78d6116942a4cd17f56d00/lib/std/fmt/parse_float/decimal.zig#L305-L309) in `std.fmt.parse_float` - [`std.http.ChunkParser.feed`](https://codeberg.org/ziglang/zig/src/commit/e5dc5a6eb5608c100e78d6116942a4cd17f56d00/lib/std/http/ChunkParser.zig#L52-L78) - [`std.zig.number_literal.parseNumberLiteral`](https://codeberg.org/ziglang/zig/src/commit/e5dc5a6eb5608c100e78d6116942a4cd17f56d00/lib/std/zig/number_literal.zig#L65-L179) - Parsing of escape sequences in strings in [zig](https://codeberg.org/ziglang/zig/src/commit/e5dc5a6eb5608c100e78d6116942a4cd17f56d00/lib/std/zig/string_literal.zig#L151-L234) and [json](https://codeberg.org/ziglang/zig/src/commit/e5dc5a6eb5608c100e78d6116942a4cd17f56d00/lib/std/json/Scanner.zig#L722-L904) - [`std.zig.AstGen.parseBitCount`](https://codeberg.org/ziglang/zig/src/commit/e5dc5a6eb5608c100e78d6116942a4cd17f56d00/lib/std/zig/AstGen.zig#L8210-L8229) - `std.process`: parsing [uid](https://codeberg.org/ziglang/zig/src/commit/e5dc5a6eb5608c100e78d6116942a4cd17f56d00/lib/std/process.zig#L199-L221) and [gid](https://codeberg.org/ziglang/zig/src/commit/e5dc5a6eb5608c100e78d6116942a4cd17f56d00/lib/std/process.zig#L222-L244) from `/etc/passwd` - [one implementation](https://codeberg.org/ziglang/zig/src/commit/e5dc5a6eb5608c100e78d6116942a4cd17f56d00/lib/c/stdlib.zig#L200-L233) used by libzigc for `ato*` and `strto*` functions (#30792). - [`std.math.big.int.Mutable.setString`](https://codeberg.org/ziglang/zig/src/commit/e5dc5a6eb5608c100e78d6116942a4cd17f56d00/lib/std/math/big/int.zig#L347-L410) - #30791 - more in aro, translate-c and resinator - maybe more that I haven't found One problem with such code duplication is that more code means more bugs. While writing this, I found #30880. Some of the implementations above (like [`std.math.big.int.Mutable.setString`](https://codeberg.org/ziglang/zig/src/commit/e5dc5a6eb5608c100e78d6116942a4cd17f56d00/lib/std/math/big/int.zig#L347-L410)) are likely too specific to benefit from a common implementation, but many of these use cases could use a common implementation. Some properties that, in my opinion, should be configurable in a common implementation (feel free to argue against these or in favour of other properties): - Should symbols like `_` be allowed between digits? Which symbols (`'`, `_`)? - Are `+` and `-` allowed? - What base is the number? Should prefixes like `0x`, `0o` and `0b` change the base to `16`, `8` and `2`. - Is a leading `0` allowed? Should it change the base to `8`? - Is the complete input string a number or only the beginning of the input string? Like the author of [#24687](https://github.com/ziglang/zig/issues/24687), I'm not sure what the ideal API would look like. [#12446](https://github.com/ziglang/zig/issues/12446) proposed a config struct. There are both use cases, for the base to be comptime known and for the base to be known only at runtime (and for the base to be specified by a prefix). For some use cases the exact length of the input string is comptime known, for other use cases it is not. For some use cases like [`std.process.posixGetUserInfoPasswdStream`](https://codeberg.org/ziglang/zig/src/commit/e5dc5a6eb5608c100e78d6116942a4cd17f56d00/lib/std/process.zig#L154-L247), it might make sense to use a `std.Io.Reader` instead of a string as the input, but in other use cases, parsing from a fixed reader would probably generate worse machine code than parsing from a string. Also, since string representations of integers are not very long in most use cases, it might make sense in general to first create a `[]const u8` of the input.

andrewrk commented 2026年01月18日 22:31:18 +01:00

Shoutouts to https://github.com/ziglang/zig/issues/19231

Shoutouts to https://github.com/ziglang/zig/issues/19231

matklad commented 2026年01月22日 13:23:48 +01:00

One problem with such code duplication is that more code means more bugs.

There's also flip side --- inappropriate code re-use could lead to accepting incorrect strings, like how we accept 6_6_6 ContentLength in HTTP: #30190.

I also stepped into this in TigerBeetle: when implementing our CLI parser, I used fmt.parseInt for numeric arguments, and I hadn't realized that it allows _ until, months later, I went to explicitly add support for _. That was a happy accident in my case, but it could have easily become permanent technical debt.

Actually, right now I realized that my CLI parser allows --cluster=-0, and that is definitely a bug: https://github.com/tigerbeetle/tigerbeetle/pull/3468

I guess my wish here is that, if we have parseInt(s) or parseIntOptions(s, .{}), I would like for those "default" forms to only parse \d+, maybe even forbidding leading zeros, and have embellishments to be explicitly opt-in.

>One problem with such code duplication is that more code means more bugs. There's also flip side --- inappropriate code re-use could lead to accepting incorrect strings, like how we accept 6_6_6 ContentLength in HTTP: https://codeberg.org/ziglang/zig/issues/30190. I also stepped into this in TigerBeetle: when implementing our CLI parser, I used `fmt.parseInt` for numeric arguments, and I hadn't realized that it allows `_` until, months later, I went to explicitly add support for `_`. That was a happy accident in my case, but it could have easily become permanent technical debt. Actually, _right now_ I realized that my CLI parser allows `--cluster=-0`, and _that_ is definitely a bug: https://github.com/tigerbeetle/tigerbeetle/pull/3468 I guess my wish here is that, if we have `parseInt(s)` or `parseIntOptions(s, .{})`, I would like for those "default" forms to only parse `\d+`, maybe even forbidding leading zeros, and have embellishments to be explicitly opt-in.

linus commented 2026年01月22日 14:25:24 +01:00

There's also flip side --- inappropriate code re-use could lead to accepting incorrect strings

I ran into this in Kiesel too, most recently in kiesel-js/kiesel@448374d53d.

I guess my wish here is that, if we have parseInt(s) or parseIntOptions(s, .{}), I would like for those "default" forms to only parse \d+, maybe even forbidding leading zeros, and have embellishments to be explicitly opt-in.

I would argue there is no obvious "good" default in this case and options should always be provided, unless the function name is sufficiently clear (e.g. parseDecimalDigits). If that's considered too verbose I agree that \d+ should be the default and everything else opt in.

---

Right now this issue is focusing on integer parsing but the same applies to std.fmt.parseFloat() - it accepts an arbitrary floating point string format that may or may not be what the application expects. IMO support for things like exponential notation should be configurable.

> There's also flip side --- inappropriate code re-use could lead to accepting incorrect strings I ran into this in Kiesel too, most recently in https://codeberg.org/kiesel-js/kiesel/commit/448374d53d4ba25c7d3db0c78b1d1c76815f92be. > I guess my wish here is that, if we have `parseInt(s)` or `parseIntOptions(s, .{})`, I would like for those "default" forms to only parse `\d+`, maybe even forbidding leading zeros, and have embellishments to be explicitly opt-in. I would argue there is no obvious "good" default in this case and options should always be provided, unless the function name is sufficiently clear (e.g. `parseDecimalDigits`). If that's considered too verbose I agree that `\d+` should be the default and everything else opt in. --- Right now this issue is focusing on integer parsing but the same applies to `std.fmt.parseFloat()` - it accepts an arbitrary floating point string format that may or may not be what the application expects. IMO support for things like exponential notation should be configurable.

GiuseppeCesarano commented 2026年03月12日 11:30:36 +01:00

I want to point out that it's clear that the intent for some of these was to optimize the implementation for a specific use case. A clear example of this is the first three functions. While I feel that specialized implementations could be reasonable in such cases, I would also argue that there are better ways to implement them, which undermines their reason to exist as "specialized for performance."

If someone with decision-making power can weigh whether to keep them or reuse some general (configurable) function, I would like to take a shot at improving things here.

I want to point out that it's clear that the intent for some of these was to optimize the implementation for a specific use case. A clear example of this is the first three functions. While I feel that specialized implementations could be reasonable in such cases, I would also argue that there are better ways to implement them, which undermines their reason to exist as "specialized for performance." If someone with decision-making power can weigh whether to keep them or reuse some general (configurable) function, I would like to take a shot at improving things here.

nektro commented 2026年03月12日 11:51:09 +01:00

I ended up landing on https://github.com/nektro/zig-extras/blob/master/src/parseDigits.zig

I ended up landing on https://github.com/nektro/zig-extras/blob/master/src/parseDigits.zig

GiuseppeCesarano commented 2026年03月12日 12:26:49 +01:00

The thing is, if the buffer length is equal to or less than the machine's @sizeOf(usize) and if no special formatting is required, SWAR techniques are usually better.

An example which reimplements the first function:

inlinefnparseTimeDigitsSWARLE(text:*const[2]u8,min:u8,max:u8)!u8{std.debug.assert(min<100);std.debug.assert(min<=max);constd:u16=@bitCast(text.*);constsub=d-%0x3030;constr:u8=@truncate((sub*%0x0A01)>>8);returnif((r-%min)<=(max-min)and(sub+|0x0606)&0xF0F0==0)relseerror.CertificateTimeInvalid;}

On my machine this is around 2.6 times faster than the one in the std.

The thing is, if the buffer length is equal to or less than the machine's `@sizeOf(usize)` and if no special formatting is required, SWAR techniques are usually better. An example which reimplements the first function: ```zig inline fn parseTimeDigitsSWARLE(text: *const [2]u8, min: u8, max: u8) !u8 { std.debug.assert(min < 100); std.debug.assert(min <= max); const d: u16 = @bitCast(text.*); const sub = d -% 0x3030; const r: u8 = @truncate((sub *% 0x0A01) >> 8); return if ((r -% min) <= (max - min) and (sub +| 0x0606) & 0xF0F0 == 0) r else error.CertificateTimeInvalid; } ``` On my machine this is around 2.6 times faster than the one in the std.