rework fuzz testing to be smith based #31205

gooncreeper commented 2026年02月14日 00:07:39 +01:00

Closes https://github.com/ziglang/zig/issues/25281
Closes https://github.com/ziglang/zig/issues/20816
Closes #30797
Migrated from https://github.com/ziglang/zig/pull/25893 (conflicts, edits)

On the standard library side:

The input: []const u8 parameter of functions passed to testing.fuzz has changed to smith: *testing.Smith. Smith is used to generate values from libfuzzer or input bytes generated by libfuzzer.

Smith contains the following base methods:

• value as a generic method for generating any type
• eos for generating end-of-stream markers. Provides the additional guarantee true will eventually by provided.
• bytes for filling a byte array.
• slice for filling part of a buffer and providing the length.

Smith.Weight is used for giving value ranges a higher probability of being selected. By default, every value has a weight of zero (i.e. they will not be selected). Weights can only apply to values that fit within a u64. The above functions have corresponding ones that accept weights. Additionally, the following functions are provided:

• baselineWeights which provides a set of weights containing every possible value of a type.
• eosSimpleWeighted for unique weights for true and false
• valueRangeAtMost and valueRangeLessThan for weighing only a range of values.

On the libfuzzer and abi side:

Uids

These are u32s which are used to classify requested values. This solves the problem of a mutation causing a new value to be requested and shifting all future values; for example:

1. An initial input contains the values 1, 2, 3 which are interpreted as a, b, and c respectively by the test.
2. The 1 is mutated to a 4 which causes the test to request an extra value interpreted as d. The input is now 4, 2, 3, 5 (new value) which the test corresponds to a, d, b, c; however, b and c no longer correspond to their original values.

Uids contain a hash component and type component. The hash component is currently determined in Smith by taking a hash of the calling @returnAddress() or via an argument in the corresponding WithHash functions. The type component is used extensively in libfuzzer with its hashmaps.

Mutations

At the start of a cycle (a run), a random number of values to mutate is selected with less being exponentially more likely. The indexes of the values are selected from a selected uid with a logarithmic bias to uids with more values.

Mutations may change a single values, several consecutive values in a uid, or several consecutive values in the uid-independent order they were requested. They may generate random values, mutate from previous ones, or copy from other values in the same uid from the same input or spliced from another.

For integers, mutations from previous ones currently only generates random values. For bytes, mutations from previous mix new random data and previous bytes with a set number of mutations.

Passive Minimization

A different approach has been taken for minimizing inputs: instead of trying a fixed set of mutations when a fresh input is found, the input is instead simply added to the corpus and removed when it is no longer valuable.

The quality of an input is measured based off how many unique pcs it hit and how many values it needed from the fuzzer. It is tracked which inputs hold the best qualities for each pc for hitting the minimum and maximum unique pcs while needing the least values.

Once all an input's qualities have been superseded for the pcs it hit, it is removed from the corpus.

Comparison to byte-based smith

A byte-based smith would be much more inefficient and complex than this solution. It would be unable to solve the shifting problem that Uids do. It is unable to provide values from the fuzzer past end-of-stream. Even with feedback, it would be unable to act on dynamic weights which have proven essential with the updated tests (e.g. to constrain values to a range).

Test updates

All the standard library tests have been updated to use the new smith interface. For Deque, an ad hoc allocator was written to improve performance and remove reliance on heap allocation. TokenSmith has been added to aid in testing Ast and help inform decisions on the smith interface.

Closes https://github.com/ziglang/zig/issues/25281 Closes https://github.com/ziglang/zig/issues/20816 Closes #30797 Migrated from https://github.com/ziglang/zig/pull/25893 (conflicts, edits) ## On the standard library side: The `input: []const u8` parameter of functions passed to `testing.fuzz` has changed to `smith: *testing.Smith`. `Smith` is used to generate values from libfuzzer or input bytes generated by libfuzzer. `Smith` contains the following base methods: * `value` as a generic method for generating any type * `eos` for generating end-of-stream markers. Provides the additional guarantee `true` will eventually by provided. * `bytes` for filling a byte array. * `slice` for filling part of a buffer and providing the length. `Smith.Weight` is used for giving value ranges a higher probability of being selected. By default, every value has a weight of zero (i.e. they will not be selected). Weights can only apply to values that fit within a u64. The above functions have corresponding ones that accept weights. Additionally, the following functions are provided: * `baselineWeights` which provides a set of weights containing every possible value of a type. * `eosSimpleWeighted` for unique weights for `true` and `false` * `valueRangeAtMost` and `valueRangeLessThan` for weighing only a range of values. ## On the libfuzzer and abi side: ### Uids These are u32s which are used to classify requested values. This solves the problem of a mutation causing a new value to be requested and shifting all future values; for example: 1. An initial input contains the values 1, 2, 3 which are interpreted as a, b, and c respectively by the test. 2. The 1 is mutated to a 4 which causes the test to request an extra value interpreted as d. The input is now 4, 2, 3, 5 (new value) which the test corresponds to a, d, b, c; however, b and c no longer correspond to their original values. Uids contain a hash component and type component. The hash component is currently determined in `Smith` by taking a hash of the calling `@returnAddress()` or via an argument in the corresponding `WithHash` functions. The type component is used extensively in libfuzzer with its hashmaps. ### Mutations At the start of a cycle (a run), a random number of values to mutate is selected with less being exponentially more likely. The indexes of the values are selected from a selected uid with a logarithmic bias to uids with more values. Mutations may change a single values, several consecutive values in a uid, or several consecutive values in the uid-independent order they were requested. They may generate random values, mutate from previous ones, or copy from other values in the same uid from the same input or spliced from another. For integers, mutations from previous ones currently only generates random values. For bytes, mutations from previous mix new random data and previous bytes with a set number of mutations. ### Passive Minimization A different approach has been taken for minimizing inputs: instead of trying a fixed set of mutations when a fresh input is found, the input is instead simply added to the corpus and removed when it is no longer valuable. The quality of an input is measured based off how many unique pcs it hit and how many values it needed from the fuzzer. It is tracked which inputs hold the best qualities for each pc for hitting the minimum and maximum unique pcs while needing the least values. Once all an input's qualities have been superseded for the pcs it hit, it is removed from the corpus. ## Comparison to byte-based smith A byte-based smith would be much more inefficient and complex than this solution. It would be unable to solve the shifting problem that Uids do. It is unable to provide values from the fuzzer past end-of-stream. Even with feedback, it would be unable to act on dynamic weights which have proven essential with the updated tests (e.g. to constrain values to a range). ## Test updates All the standard library tests have been updated to use the new smith interface. For `Deque`, an ad hoc allocator was written to improve performance and remove reliance on heap allocation. `TokenSmith` has been added to aid in testing Ast and help inform decisions on the smith interface.

andrewrk commented 2026年02月14日 00:15:11 +01:00

Thanks for refreshing this

Thanks for refreshing this

grimtyr commented 2026年02月14日 01:55:23 +01:00

Hi thanks for these changes, I saw in file zig/lib/std/deque.zig line 661 and a few lines after there is still reference to random that should be smith maybe

Hi thanks for these changes, I saw in file `zig/lib/std/deque.zig` line 661 and a few lines after there is still reference to random that should be smith maybe

gooncreeper commented 2026年02月14日 04:14:05 +01:00

@grimtyr wrote in #31205 (comment):

Hi thanks for these changes, I saw in file zig/lib/std/deque.zig line 661 and a few lines after there is still reference to random that should be smith maybe

Thanks, I updated it.

@grimtyr wrote in https://codeberg.org/ziglang/zig/pulls/31205#issuecomment-10571070: > Hi thanks for these changes, I saw in file `zig/lib/std/deque.zig` line 661 and a few lines after there is still reference to random that should be smith maybe Thanks, I updated it.

grimtyr commented 2026年02月14日 04:21:30 +01:00

I tried the zig init default fuzz test, But it gives an error, I don't know if that was intended or if its a bug, I tried to reproduce it normally with this code but It's able to resize the array normally while the fuzz test cannot:

pubfnmain(init:std.process.Init)!void{constgpa=init.gpa;varlist:std.ArrayList(u8)=.empty;deferlist.deinit(gpa);varslice=trylist.addManyAsSlice(gpa,108);fillBytes(slice);slice=trylist.addManyAsSlice(gpa,18);fillBytes(slice);slice=trylist.addManyAsSlice(gpa,8);fillBytes(slice);constlen=14;constoff=83;constitems=list.items[off..][0..len];std.debug.print("llen:len:off:arr\n",.{});std.debug.print("{d:>4}:{d:>3}:{d:>3}:{any:>3}:{any:>3}\n",.{list.items.len,len,off,list.items[off..][0..len],list.items[list.items.len-len..],});trylist.appendSlice(gpa,items);}

fuzz test:

fntestOne(context:void,smith:*std.testing.Smith)!void{_=context;// Try passing `--fuzz` to `zig build test` and see if it manages to fail this test case!constgpa=std.testing.allocator;varlist:std.ArrayList(u8)=.empty;deferlist.deinit(gpa);while(!smith.eos())switch(smith.value(enum{add_data,dup_data})){.add_data=>{constslice=trylist.addManyAsSlice(gpa,smith.value(u4));smith.bytes(slice);},.dup_data=>{if(list.items.len==0)continue;if(list.items.len>std.math.maxInt(u32))returnerror.SkipZigTest;constlen=smith.valueRangeAtMost(u32,1,@min(32,list.items.len));constoff=smith.valueRangeAtMost(u32,0,@intCast(list.items.len-len));trylist.appendSlice(gpa,list.items[off..][0..len]);trystd.testing.expectEqualSlices(u8,list.items[off..][0..len],list.items[list.items.len-len..],);},};}

The error:

test
└─ run test failure
Segmentation fault at address 0x7f73bd160053
/home/grimtyr/personal/src/zig-gt/lib/compiler_rt/memcpy.zig:170:17: 0x129a922 in copyFixedLength (compiler_rt)
 d[i] = s[i];
 ^
/home/grimtyr/personal/src/zig-gt/lib/std/array_list.zig:985:42: 0x11cdee5 in appendSliceAssumeCapacity (std.zig)
 @memcpy(self.items[old_len..][0..items.len], items);
 ^
/home/grimtyr/personal/src/zig-gt/lib/std/array_list.zig:974:43: 0x11ca7ff in appendSlice (std.zig)
 self.appendSliceAssumeCapacity(items);
 ^
/tmp/tmp.mCQ89m2R6Y/src/main.zig:113:33: 0x11c96f1 in testOne (main.zig)
 try list.appendSlice(gpa, list.items[off..][0..len]);
 ^
/home/grimtyr/personal/src/zig-gt/lib/compiler/test_runner.zig:414:20: 0x11c53fb in test_one (test_runner.zig)
 testOne(ctx, @constCast(&testing.Smith{ .in = null })) catch |err| switch (err) {
 ^
/home/grimtyr/personal/src/zig-gt/lib/fuzzer.zig:1006:19: 0x12644aa in run (fuzzer)
 f.test_one();
 ^
/home/grimtyr/personal/src/zig-gt/lib/compiler/test_runner.zig:440:33: 0x11c5064 in fuzz__anon_28825 (test_runner.zig)
 fuzz_abi.fuzzer_set_test(&global.test_one, .fromSlice(builtin.test_functions[fuzz_test_index].name));
 ^
/home/grimtyr/personal/src/zig-gt/lib/std/testing.zig:1218:32: 0x11c4dad in fuzz (main.zig)
 return @import("root").fuzz(context, testOne, options);
 ^
/home/grimtyr/personal/src/zig-gt/lib/compiler/test_runner.zig:197:29: 0x11f8dab in mainServer (test_runner.zig)
 test_fn.func() catch |err| switch (err) {
 ^
/home/grimtyr/personal/src/zig-gt/lib/compiler/test_runner.zig:68:26: 0x11f9e10 in main (test_runner.zig)
 return mainServer(init) catch @panic("internal test runner failure");
 ^
/home/grimtyr/personal/src/zig-gt/lib/std/start.zig:678:88: 0x11f4ec6 in callMain (std.zig)
 if (fn_info.params[0].type.? == std.process.Init.Minimal) return wrapMain(root.main(.{
 ^
/home/grimtyr/personal/src/zig-gt/lib/std/start.zig:190:5: 0x11f48d1 in _start (std.zig)
 asm volatile (switch (native_arch) {
 ^
error: test process unexpectedly terminated with signal ABRT
failed command: ./.zig-cache/o/31f8e5f39c60eac47c06b104bc067bea/test --cache-dir=./.zig-cache --seed=0x5f5b0eaf --listen=-
======= FUZZING REPORT =======
error: step 'run test': corrupted coverage file '.zig-cache/v/59e4bde28e630904': pcs_len was zero
error: the following build command failed with exit code 1:
.zig-cache/o/f8784b54cbaed93e4a12c6a611e7a16c/build /home/grimtyr/personal/src/zig-gt/build/stage4/bin/zig /home/grimtyr/personal/src/zig-gt/lib /tmp/tmp.mCQ89m2R6Y .zig-cache /home/grimtyr/.cache/zig --seed 0x5f5b0eaf -Z94002b71308a2e70 test --fuzz=100K

I tried the `zig init` default fuzz test, But it gives an error, I don't know if that was intended or if its a bug, I tried to reproduce it normally with this code but It's able to resize the array normally while the fuzz test cannot: ```zig pub fn main(init: std.process.Init) !void { const gpa = init.gpa; var list: std.ArrayList(u8) = .empty; defer list.deinit(gpa); var slice = try list.addManyAsSlice(gpa, 108); fillBytes(slice); slice = try list.addManyAsSlice(gpa, 18); fillBytes(slice); slice = try list.addManyAsSlice(gpa, 8); fillBytes(slice); const len = 14; const off = 83; const items = list.items[off..][0..len]; std.debug.print("llen:len:off:arr\n", .{}); std.debug.print("{d:>4}:{d:>3}:{d:>3}:{any:>3}:{any:>3}\n", .{ list.items.len, len, off, list.items[off..][0..len], list.items[list.items.len - len ..], }); try list.appendSlice(gpa, items); } ``` fuzz test: ```zig fn testOne(context: void, smith: *std.testing.Smith) !void { _ = context; // Try passing `--fuzz` to `zig build test` and see if it manages to fail this test case! const gpa = std.testing.allocator; var list: std.ArrayList(u8) = .empty; defer list.deinit(gpa); while (!smith.eos()) switch (smith.value(enum { add_data, dup_data })) { .add_data => { const slice = try list.addManyAsSlice(gpa, smith.value(u4)); smith.bytes(slice); }, .dup_data => { if (list.items.len == 0) continue; if (list.items.len > std.math.maxInt(u32)) return error.SkipZigTest; const len = smith.valueRangeAtMost(u32, 1, @min(32, list.items.len)); const off = smith.valueRangeAtMost(u32, 0, @intCast(list.items.len - len)); try list.appendSlice(gpa, list.items[off..][0..len]); try std.testing.expectEqualSlices( u8, list.items[off..][0..len], list.items[list.items.len - len ..], ); }, }; } ``` The error: ```shell test └─ run test failure Segmentation fault at address 0x7f73bd160053 /home/grimtyr/personal/src/zig-gt/lib/compiler_rt/memcpy.zig:170:17: 0x129a922 in copyFixedLength (compiler_rt) d[i] = s[i]; ^ /home/grimtyr/personal/src/zig-gt/lib/std/array_list.zig:985:42: 0x11cdee5 in appendSliceAssumeCapacity (std.zig) @memcpy(self.items[old_len..][0..items.len], items); ^ /home/grimtyr/personal/src/zig-gt/lib/std/array_list.zig:974:43: 0x11ca7ff in appendSlice (std.zig) self.appendSliceAssumeCapacity(items); ^ /tmp/tmp.mCQ89m2R6Y/src/main.zig:113:33: 0x11c96f1 in testOne (main.zig) try list.appendSlice(gpa, list.items[off..][0..len]); ^ /home/grimtyr/personal/src/zig-gt/lib/compiler/test_runner.zig:414:20: 0x11c53fb in test_one (test_runner.zig) testOne(ctx, @constCast(&testing.Smith{ .in = null })) catch |err| switch (err) { ^ /home/grimtyr/personal/src/zig-gt/lib/fuzzer.zig:1006:19: 0x12644aa in run (fuzzer) f.test_one(); ^ /home/grimtyr/personal/src/zig-gt/lib/compiler/test_runner.zig:440:33: 0x11c5064 in fuzz__anon_28825 (test_runner.zig) fuzz_abi.fuzzer_set_test(&global.test_one, .fromSlice(builtin.test_functions[fuzz_test_index].name)); ^ /home/grimtyr/personal/src/zig-gt/lib/std/testing.zig:1218:32: 0x11c4dad in fuzz (main.zig) return @import("root").fuzz(context, testOne, options); ^ /home/grimtyr/personal/src/zig-gt/lib/compiler/test_runner.zig:197:29: 0x11f8dab in mainServer (test_runner.zig) test_fn.func() catch |err| switch (err) { ^ /home/grimtyr/personal/src/zig-gt/lib/compiler/test_runner.zig:68:26: 0x11f9e10 in main (test_runner.zig) return mainServer(init) catch @panic("internal test runner failure"); ^ /home/grimtyr/personal/src/zig-gt/lib/std/start.zig:678:88: 0x11f4ec6 in callMain (std.zig) if (fn_info.params[0].type.? == std.process.Init.Minimal) return wrapMain(root.main(.{ ^ /home/grimtyr/personal/src/zig-gt/lib/std/start.zig:190:5: 0x11f48d1 in _start (std.zig) asm volatile (switch (native_arch) { ^ error: test process unexpectedly terminated with signal ABRT failed command: ./.zig-cache/o/31f8e5f39c60eac47c06b104bc067bea/test --cache-dir=./.zig-cache --seed=0x5f5b0eaf --listen=- ======= FUZZING REPORT ======= error: step 'run test': corrupted coverage file '.zig-cache/v/59e4bde28e630904': pcs_len was zero error: the following build command failed with exit code 1: .zig-cache/o/f8784b54cbaed93e4a12c6a611e7a16c/build /home/grimtyr/personal/src/zig-gt/build/stage4/bin/zig /home/grimtyr/personal/src/zig-gt/lib /tmp/tmp.mCQ89m2R6Y .zig-cache /home/grimtyr/.cache/zig --seed 0x5f5b0eaf -Z94002b71308a2e70 test --fuzz=100K ```

gooncreeper commented 2026年02月14日 15:50:06 +01:00

@grimtyr wrote in #31205 (comment):

I tried the zig init default fuzz test, But it gives an error, I don't know if that was intended or if its a bug, I tried to reproduce it normally with this code but It's able to resize the array normally while the fuzz test cannot:

The bug is intentional (see if it manages to fail this test case). It is a bit subtle but the slice passed in try list.appendSlice(gpa, list.items[off..][0..len]); is referencing invalidated memory since list is resized. The point is the fuzz test is exposing a bug that is not immediately obvious and requires verily specific conditions.

@grimtyr wrote in https://codeberg.org/ziglang/zig/pulls/31205#issuecomment-10571798: > I tried the `zig init` default fuzz test, But it gives an error, I don't know if that was intended or if its a bug, I tried to reproduce it normally with this code but It's able to resize the array normally while the fuzz test cannot: The bug is intentional (`see if it manages to fail this test case`). It is a bit subtle but the slice passed in `try list.appendSlice(gpa, list.items[off..][0..len]);` is referencing invalidated memory since `list` is resized. The point is the fuzz test is exposing a bug that is not immediately obvious and requires verily specific conditions.

grimtyr commented 2026年02月14日 16:11:10 +01:00

Oh yeah that makes sense, It's my bad I didn't check what the capacity was and I though it did not matter, didn't catch it in the list.addManyAsSlice doc comment, thanks again for this and your help.

Oh yeah that makes sense, It's my bad I didn't check what the capacity was and I though it did not matter, didn't catch it in the `list.addManyAsSlice` doc comment, thanks again for this and your help.

Cajunvoodoo commented 2026年03月24日 18:42:58 +01:00

Just curious, is there anything about FuzzAllocator that requires it to be in lib/std/deque.zig specifically, or is it more generally applicable to other uses of Smith? I can't tell by looking at it what is specific to deques.

Just curious, is there anything about `FuzzAllocator` that requires it to be in `lib/std/deque.zig` specifically, or is it more generally applicable to other uses of `Smith`? I can't tell by looking at it what is specific to deques.

gooncreeper commented 2026年03月24日 21:20:11 +01:00

@Cajunvoodoo wrote in #31205 (comment):

Just curious, is there anything about FuzzAllocator that requires it to be in lib/std/deque.zig specifically, or is it more generally applicable to other uses of Smith? I can't tell by looking at it what is specific to deques.

Yes, it asserts a maximum of two allocations at any time, allocations are asserted to have an alignment of 4, allocation resizes assert only one active allocation, and the maximum allocation size is fixed. These match the fuzz test as it only uses u32 and match deques since they should only have one allocation, except two when resizing.

@Cajunvoodoo wrote in https://codeberg.org/ziglang/zig/pulls/31205#issuecomment-12033456: > Just curious, is there anything about `FuzzAllocator` that requires it to be in `lib/std/deque.zig` specifically, or is it more generally applicable to other uses of `Smith`? I can't tell by looking at it what is specific to deques. Yes, it asserts a maximum of two allocations at any time, allocations are asserted to have an alignment of 4, allocation resizes assert only one active allocation, and the maximum allocation size is fixed. These match the fuzz test as it only uses u32 and match deques since they should only have one allocation, except two when resizing.