stdout output should be cleaned after successful tests #35958

Zig Version

0.16.0

Steps to Reproduce, Observed Behavior, and Expected Behavior

A failing test after a successful test gets all the print msgs that get printed during the successful test.
i.e.

const std = @import("std");
test "foo" {
 std.debug.print("foo\n", .{});
}
test "bar" {
 return error.Error;
}

and run zig build test with the default init.
(maybe andrew fixed this in the build process refactor; I didn't check master)

Edit: (ty @K4)

Observed Behavior

test
└─ run test 1 pass, 1 fail (2 total)
error: 'main.test.bar' failed:
 foo
 /home/user/code/personal-projects/testingzigbugs/a/src/main.zig:7:5: 0x1249f8c in test.bar (main.zig)
 return error.Error;
 ^
failed command: ./.zig-cache/o/b08b615b2388f36bf926a7724608ccbc/test --cache-dir=./.zig-cache --seed=0x950f923 --listen=-
Build Summary: 3/5 steps succeeded (1 failed); 2/3 tests passed (1 failed)
test transitive failure
└─ run test 1 pass, 1 fail (2 total)
error: the following maker command exited with code 1:
/home/user/.cache/zig/o/debddda5b2a68b28d44b8dec9840ce63/maker --zig /home/user/.zig-master/zig --zig-lib-dir /home/user/.zig-master/lib --build-root /home/user/code/personal-projects/testingzigbugs/a --local-cache .zig-cache --global-cache /home/user/.cache/zig --configuration .zig-cache/c/746407df73b3e8d3761d82d6b43a1e94 --seed 0x950f923 test

Expected Behavoir

(this is how it works when using zig test, it just prints it regardless of whether the test fails or not)

foo
test
└─ run test 1 pass, 1 fail (2 total)
error: 'main.test.bar' failed:
 /home/user/code/personal-projects/testingzigbugs/a/src/main.zig:7:5: 0x1249f8c in test.bar (main.zig)
 return error.Error;
 ^
failed command: ./.zig-cache/o/b08b615b2388f36bf926a7724608ccbc/test --cache-dir=./.zig-cache --seed=0x950f923 --listen=-
Build Summary: 3/5 steps succeeded (1 failed); 2/3 tests passed (1 failed)
test transitive failure
└─ run test 1 pass, 1 fail (2 total)
error: the following maker command exited with code 1:
/home/user/.cache/zig/o/debddda5b2a68b28d44b8dec9840ce63/maker --zig /home/user/.zig-master/zig --zig-lib-dir /home/user/.zig-master/lib --build-root /home/user/code/personal-projects/testingzigbugs/a --local-cache .zig-cache --global-cache /home/user/.cache/zig --configuration .zig-cache/c/746407df73b3e8d3761d82d6b43a1e94 --seed 0x950f923 test

missing Observed Behavior

I have submitted a pull request (#36057) to fix stdout/stderr contamination in the test runner.

The fix works as expected with the pre-compiled Zig 0.16.0 binary. However, it causes CI build failures in some environments, where incompatible pointer types warnings are being treated as errors.

I am reporting this to document that the fix is logically correct, but it currently encounters build issues in some CI environments.

$ zig version
0.16.0
$ zig build test 
test
└─ run test 1 pass, 1 fail (2 total)
error: 'main.test.bar' failed:
 /home/user/projects/contrib/src/main.zig:7:5: 0x12388ec in test.bar (main.zig)
 return error.Error;
 ^
failed command: /home/user/projects/contrib/.zig-cache/o/cb5f50dd65e052cbb7d2790ac8a4cd22/test --cache-dir=/home/user/projects/contrib/.zig-cache --seed=0x85bd233 --listen=-
Build Summary: 3/5 steps succeeded (1 failed); 2/3 tests passed (1 failed)
test transitive failure
└─ run test 1 pass, 1 fail (2 total)
error: the following build command failed with exit code 1:
/home/user/projects/contrib/.zig-cache/o/e3797b83550f144175d6b840dfc11fe3/build /home/user/Downloads/zig-x86_64-linux-0.16.0/zig /home/user/Downloads/zig-x86_64-linux-0.16.0/lib /home/user/projects/contrib /home/user/projects/contrib/.zig-cache /home/user/.cache/zig --seed 0x85bd233 -Z4fbf69856b8baba4 test