ziglang/zig
148
2.9k
Fork
You've already forked zig
272

std.Io.Threaded: debug file handle leaks #30788

Open
andrewrk wants to merge 1 commit from debug-file-leaks into master
pull from: debug-file-leaks
merge into: ziglang:master
ziglang:master
ziglang:mmap
ziglang:riscv-ci-2
ziglang:riscv-ci
ziglang:test-no-bin
ziglang:poll
ziglang:io-uring-update
ziglang:llvm22
ziglang:poll-ring
ziglang:debug-file-leaks-differently
ziglang:hate-letter-to-std.os
ziglang:i-am-a-foolish-fool
ziglang:ProcessPrng
ziglang:elfv2-dyn
ziglang:jobserver
ziglang:threadtheft
ziglang:io-threaded-no-queue
ziglang:0.15.x
ziglang:Io.net
ziglang:comptime-allocator
ziglang:restricted-function-pointers
ziglang:cli
ziglang:wasm-linker-writer
ziglang:wrangle-writer-buffering
ziglang:sha1-stream
ziglang:async-await-demo
ziglang:fixes
ziglang:0.14.x
ziglang:ast-node-methods
ziglang:spork8
ziglang:macos-debug-info
ziglang:make-vs-configure
ziglang:fuzz-macos
ziglang:main
ziglang:sans-aro
ziglang:ArrayList-reserve
ziglang:incr-bug
ziglang:llvm-ir-nosanitize-metadata
ziglang:ci-tarballs
ziglang:ci-scripts
ziglang:threadpool
ziglang:0.12.x
ziglang:new-pkg-hash
ziglang:json-diagnostics
ziglang:more-doctests
ziglang:rework-comptime-mutation
ziglang:0.11.x
ziglang:ci-perf-comment
ziglang:stage2-async
ziglang:0.10.x
ziglang:autofix
ziglang:0.9.x
ziglang:aro
ziglang:hcs
ziglang:0.8.x
ziglang:0.7.x

As promised, one of the neat features of std.Io is that it makes it nice to debug resource leaks such as file descriptors. Here is a proof-of-concept of this working:

Demo

conststd=@import("std");constIo=std.Io;pubfnmain(init:std.process.Init)!void{constio=init.io;constargs=tryinit.minimal.args.toSlice(init.arena.allocator());for(args[1..])|arg|{constfile=tryIo.Dir.cwd().openFile(io,arg,.{});_=file;}}
$ zig run test.zig -- hello.zig windows_h.zig 
error: file handle 3 leaked: 
/home/andy/src/zig/lib/std/Io/Dir.zig:508:33: 0x10cd890 in openFile (std.zig)
 return io.vtable.dirOpenFile(io.userdata, dir, sub_path, flags);
 ^
/home/andy/src/zig/build-release/test.zig:9:47: 0x11ae0f5 in main (test.zig)
 const file = try Io.Dir.cwd().openFile(io, arg, .{});
 ^
/home/andy/src/zig/lib/std/start.zig:714:30: 0x11aebe6 in callMain (std.zig)
 return wrapMain(root.main(.{
 ^
/home/andy/src/zig/lib/std/start.zig:190:5: 0x11ade51 in _start (std.zig)
 asm volatile (switch (native_arch) {
 ^
???:?:?: 0x0 in ??? (???)
???:?:?: 0x0 in ??? (???)
error: file handle 4 leaked: 
/home/andy/src/zig/lib/std/Io/Dir.zig:508:33: 0x10cd890 in openFile (std.zig)
 return io.vtable.dirOpenFile(io.userdata, dir, sub_path, flags);
 ^
/home/andy/src/zig/build-release/test.zig:9:47: 0x11ae0f5 in main (test.zig)
 const file = try Io.Dir.cwd().openFile(io, arg, .{});
 ^
/home/andy/src/zig/lib/std/start.zig:714:30: 0x11aebe6 in callMain (std.zig)
 return wrapMain(root.main(.{
 ^
/home/andy/src/zig/lib/std/start.zig:190:5: 0x11ade51 in _start (std.zig)
 asm volatile (switch (native_arch) {
 ^
???:?:?: 0x0 in ??? (???)
???:?:?: 0x0 in ??? (???)
As promised, one of the neat features of `std.Io` is that it makes it nice to debug resource leaks such as file descriptors. Here is a proof-of-concept of this working: ## Demo ```zig const std = @import("std"); const Io = std.Io; pub fn main(init: std.process.Init) !void { const io = init.io; const args = try init.minimal.args.toSlice(init.arena.allocator()); for (args[1..]) |arg| { const file = try Io.Dir.cwd().openFile(io, arg, .{}); _ = file; } } ``` ``` $ zig run test.zig -- hello.zig windows_h.zig error: file handle 3 leaked: /home/andy/src/zig/lib/std/Io/Dir.zig:508:33: 0x10cd890 in openFile (std.zig) return io.vtable.dirOpenFile(io.userdata, dir, sub_path, flags); ^ /home/andy/src/zig/build-release/test.zig:9:47: 0x11ae0f5 in main (test.zig) const file = try Io.Dir.cwd().openFile(io, arg, .{}); ^ /home/andy/src/zig/lib/std/start.zig:714:30: 0x11aebe6 in callMain (std.zig) return wrapMain(root.main(.{ ^ /home/andy/src/zig/lib/std/start.zig:190:5: 0x11ade51 in _start (std.zig) asm volatile (switch (native_arch) { ^ ???:?:?: 0x0 in ??? (???) ???:?:?: 0x0 in ??? (???) error: file handle 4 leaked: /home/andy/src/zig/lib/std/Io/Dir.zig:508:33: 0x10cd890 in openFile (std.zig) return io.vtable.dirOpenFile(io.userdata, dir, sub_path, flags); ^ /home/andy/src/zig/build-release/test.zig:9:47: 0x11ae0f5 in main (test.zig) const file = try Io.Dir.cwd().openFile(io, arg, .{}); ^ /home/andy/src/zig/lib/std/start.zig:714:30: 0x11aebe6 in callMain (std.zig) return wrapMain(root.main(.{ ^ /home/andy/src/zig/lib/std/start.zig:190:5: 0x11ade51 in _start (std.zig) asm volatile (switch (native_arch) { ^ ???:?:?: 0x0 in ??? (???) ???:?:?: 0x0 in ??? (???) ```
std.Io.Threaded: debug file handle leaks
Some checks failed
ci / aarch64-linux-release (pull_request) Failing after 14m30s
ci / aarch64-linux-debug (pull_request) Failing after 16m28s
ci / x86_64-freebsd-debug (pull_request) Failing after 11m40s
ci / x86_64-freebsd-release (pull_request) Failing after 7m24s
ci / x86_64-windows-release (pull_request) Failing after 3m29s
ci / x86_64-windows-debug (pull_request) Failing after 5m19s
ci / x86_64-linux-debug (pull_request) Failing after 12m34s
ci / aarch64-macos-debug (pull_request) Failing after 18m43s
ci / x86_64-linux-debug-llvm (pull_request) Failing after 12m7s
ci / x86_64-openbsd-debug (pull_request) Failing after 12m11s
ci / x86_64-linux-release (pull_request) Failing after 11m24s
ci / x86_64-openbsd-release (pull_request) Failing after 7m10s
ci / aarch64-macos-release (pull_request) Failing after 16m4s
ci / loongarch64-linux-debug (pull_request) Failing after 24m26s
ci / loongarch64-linux-release (pull_request) Failing after 21m42s
ci / s390x-linux-debug (pull_request) Failing after 12m27s
ci / s390x-linux-release (pull_request) Failing after 11m40s
ci / powerpc64le-linux-debug (pull_request) Failing after 17m35s
ci / powerpc64le-linux-release (pull_request) Failing after 16m22s
479e62e8ee
Owner
Copy link

I've pushed a counter-proposal to the branch debug-file-leaks-differently (link to commit). There are a few small tweaks in the implementation, but that's not really important: the difference I want to discuss is that instead of hardcoding this logic into Io.Threaded, I've instead written it as a separate implementation, Io.Debug, which wraps another Io (like how std.heap.ArenaAllocator wraps another Allocator). I think this might be a better idea for a few reasons:

  • The implementation is more readable. Most obviously, this PR makes the big Io.Threaded even bigger. But another downside of the approach you've gone with is that you've had to put calls to trackOpenFile separately in the code paths for every target (dirOpenFilePosix, dirOpenFileWtf16, etc)---there's no need for that mistake (missing the call on some target[s]) to even be possible!
  • It allows you to trivially plug in different implementations to this debugging framework. I think this is a good idea for two reasons:
    • Evented may have significantly better performance characteristics for some applications, so even in debug mode with Threaded available, some users may prefer to use Evented.
    • On some targets, Threaded may not be an available implementation. For instance, WASM in the browser might have no threads available to it, and choose to use an Io implementation based on stackless coroutines. You still want nice debugging capabilities in that case! Of course, there's also freestanding development / BYOS, where you need to write your own Io implementation.

EDIT: oh, and just to be clear, the implementation behaves just the same as yours on your example:

$ zig run repro.zig -- README.md CMakeLists.txt
error: file handle '3' leaked:
/home/mlugg/zig/master/lib/std/Io/Dir.zig:508:33: 0x10ebfb0 in openFile (std.zig)
 return io.vtable.dirOpenFile(io.userdata, dir, sub_path, flags);
 ^
/home/mlugg/zig/master/repro.zig:6:47: 0x11c7a1e in main (repro.zig)
 const file = try Io.Dir.cwd().openFile(io, arg, .{});
 ^
/home/mlugg/zig/master/lib/std/start.zig:728:30: 0x11c85f7 in callMain (std.zig)
 return wrapMain(root.main(.{
 ^
/home/mlugg/zig/master/lib/std/start.zig:190:5: 0x11c7751 in _start (std.zig)
 asm volatile (switch (native_arch) {
 ^
error: file handle '4' leaked:
/home/mlugg/zig/master/lib/std/Io/Dir.zig:508:33: 0x10ebfb0 in openFile (std.zig)
 return io.vtable.dirOpenFile(io.userdata, dir, sub_path, flags);
 ^
/home/mlugg/zig/master/repro.zig:6:47: 0x11c7a1e in main (repro.zig)
 const file = try Io.Dir.cwd().openFile(io, arg, .{});
 ^
/home/mlugg/zig/master/lib/std/start.zig:728:30: 0x11c85f7 in callMain (std.zig)
 return wrapMain(root.main(.{
 ^
/home/mlugg/zig/master/lib/std/start.zig:190:5: 0x11c7751 in _start (std.zig)
 asm volatile (switch (native_arch) {
 ^
I've pushed a counter-proposal to the branch `debug-file-leaks-differently` ([link to commit](https://codeberg.org/ziglang/zig/commit/fa74b80ab6c4932d10fffd267d88da4a3c01b4a4)). There are a few small tweaks in the implementation, but that's not really important: the difference I want to discuss is that instead of hardcoding this logic into `Io.Threaded`, I've instead written it as a separate implementation, `Io.Debug`, which wraps another `Io` (like how `std.heap.ArenaAllocator` wraps another `Allocator`). I think this might be a better idea for a few reasons: * The implementation is more readable. Most obviously, this PR makes the big `Io.Threaded` even bigger. But another downside of the approach you've gone with is that you've had to put calls to `trackOpenFile` separately in the code paths for every target (`dirOpenFilePosix`, `dirOpenFileWtf16`, etc)---there's no need for that mistake (missing the call on some target[s]) to even be possible! * It allows you to trivially plug in different implementations to this debugging framework. I think this is a good idea for two reasons: * `Evented` may have *significantly* better performance characteristics for some applications, so even in debug mode with `Threaded` available, some users may prefer to use `Evented`. * On some targets, `Threaded` may not be an available implementation. For instance, WASM in the browser might have no threads available to it, and choose to use an `Io` implementation based on stackless coroutines. You still want nice debugging capabilities in that case! Of course, there's also freestanding development / BYOS, where you need to write your own `Io` implementation. --- EDIT: oh, and just to be clear, the implementation behaves just the same as yours on your example: ``` $ zig run repro.zig -- README.md CMakeLists.txt error: file handle '3' leaked: /home/mlugg/zig/master/lib/std/Io/Dir.zig:508:33: 0x10ebfb0 in openFile (std.zig) return io.vtable.dirOpenFile(io.userdata, dir, sub_path, flags); ^ /home/mlugg/zig/master/repro.zig:6:47: 0x11c7a1e in main (repro.zig) const file = try Io.Dir.cwd().openFile(io, arg, .{}); ^ /home/mlugg/zig/master/lib/std/start.zig:728:30: 0x11c85f7 in callMain (std.zig) return wrapMain(root.main(.{ ^ /home/mlugg/zig/master/lib/std/start.zig:190:5: 0x11c7751 in _start (std.zig) asm volatile (switch (native_arch) { ^ error: file handle '4' leaked: /home/mlugg/zig/master/lib/std/Io/Dir.zig:508:33: 0x10ebfb0 in openFile (std.zig) return io.vtable.dirOpenFile(io.userdata, dir, sub_path, flags); ^ /home/mlugg/zig/master/repro.zig:6:47: 0x11c7a1e in main (repro.zig) const file = try Io.Dir.cwd().openFile(io, arg, .{}); ^ /home/mlugg/zig/master/lib/std/start.zig:728:30: 0x11c85f7 in callMain (std.zig) return wrapMain(root.main(.{ ^ /home/mlugg/zig/master/lib/std/start.zig:190:5: 0x11c7751 in _start (std.zig) asm volatile (switch (native_arch) { ^ ```
Some checks failed
ci / aarch64-linux-release (pull_request) Failing after 14m30s
Required
Details
ci / aarch64-linux-debug (pull_request) Failing after 16m28s
Required
Details
ci / x86_64-freebsd-debug (pull_request) Failing after 11m40s
Required
Details
ci / x86_64-freebsd-release (pull_request) Failing after 7m24s
Required
Details
ci / x86_64-windows-release (pull_request) Failing after 3m29s
Required
Details
ci / x86_64-windows-debug (pull_request) Failing after 5m19s
Required
Details
ci / x86_64-linux-debug (pull_request) Failing after 12m34s
Required
Details
ci / aarch64-macos-debug (pull_request) Failing after 18m43s
Required
Details
ci / x86_64-linux-debug-llvm (pull_request) Failing after 12m7s
Required
Details
ci / x86_64-openbsd-debug (pull_request) Failing after 12m11s
Required
Details
ci / x86_64-linux-release (pull_request) Failing after 11m24s
Required
Details
ci / x86_64-openbsd-release (pull_request) Failing after 7m10s
Required
Details
ci / aarch64-macos-release (pull_request) Failing after 16m4s
Required
Details
ci / loongarch64-linux-debug (pull_request) Failing after 24m26s
ci / loongarch64-linux-release (pull_request) Failing after 21m42s
ci / s390x-linux-debug (pull_request) Failing after 12m27s
ci / s390x-linux-release (pull_request) Failing after 11m40s
ci / powerpc64le-linux-debug (pull_request) Failing after 17m35s
ci / powerpc64le-linux-release (pull_request) Failing after 16m22s
Some required checks were not successful.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin debug-file-leaks:debug-file-leaks
git switch debug-file-leaks
Sign in to join this conversation.
No reviewers
Labels
Clear labels
abi/f32
abi/ilp32
abi/n32
abi/sf
abi/x32
accepted

This proposal is planned.
arch/1750a
arch/21k
arch/6502
arch/a29k
arch/aarch64
arch/alpha
arch/amdgcn
arch/arc
arch/arc32
arch/arc64
arch/arm
arch/avr
arch/avr32
arch/bfin
arch/bpf
arch/clipper
arch/colossus
arch/cr16
arch/cris
arch/csky
arch/dlx
arch/dsp16xx
arch/elxsi
arch/epiphany
arch/fr30
arch/frv
arch/h8300
arch/h8500
arch/hexagon
arch/hppa
arch/hppa64
arch/i370
arch/i860
arch/i960
arch/ia64
arch/ip2k
arch/kalimba
arch/kvx
arch/lanai
arch/lm32
arch/loongarch32
arch/loongarch64
arch/m32r
arch/m68k
arch/m88k
arch/maxq
arch/mcore
arch/metag
arch/microblaze
arch/mips
arch/mips64
arch/mmix
arch/mn10200
arch/mn10300
arch/moxie
arch/mrisc32
arch/msp430
arch/nds32
arch/nios2
arch/ns32k
arch/nvptx
arch/or1k
arch/pdp10
arch/pdp11
arch/pj
arch/powerpc
arch/powerpc64
arch/propeller
arch/riscv32
arch/riscv64
arch/rl78
arch/rx
arch/s390
arch/s390x
arch/sh
arch/sh64
arch/sparc
arch/sparc64
arch/spirv
arch/spu
arch/st200
arch/starcore
arch/tilegx
arch/tilepro
arch/tricore
arch/ts
arch/v850
arch/vax
arch/vc4
arch/ve
arch/wasm
arch/we32k
arch/x86
arch/x86_16
arch/x86_64
arch/xcore
arch/xgate
arch/xstormy16
arch/xtensa
autodoc

The web application for interactive documentation and generation of its assets.
backend/c

The C backend outputs C source code.
backend/llvm

The LLVM backend outputs an LLVM bitcode module.
backend/self-hosted

The self-hosted backends produce machine code directly.
binutils

Zig's included binary utilities: zig ar, zig dlltool, zig lib, zig ranlib, zig objcopy, and zig rc.
breaking

Implementing this issue could cause existing code to no longer compile or have different behavior.
build system

The Zig build system - zig build, std.Build, the build runner, and package management.
debug info

An issue related to debug information (e.g. DWARF) produced by the Zig compiler.
docs

An issue with documentation, e.g. the language reference or standard library doc comments.
error message

This issue points out an error message that is unhelpful and should be improved.
frontend

Tokenization, parsing, AstGen, ZonGen, Sema, Legalize, and Liveness.
fuzzing

An issue related to Zig's integrated fuzz testing.
incremental

Reuse of internal compiler state for faster compilation.
lib/c

This issue relates to Zig's libc implementation and/or vendored libcs.
lib/compiler-rt

This issue relates to Zig's compiler-rt library.
lib/cxx

This issue relates to Zig's vendored libc++ and/or libc++abi.
lib/std

This issue relates to Zig's standard library.
lib/tsan

This issue relates to Zig's vendored libtsan.
lib/ubsan-rt

This issue relates to Zig's ubsan-rt library.
lib/unwind

This issue relates to Zig's vendored libunwind.
linking

Zig's integrated object file and incremental linker.
miscompilation

The compiler reports success but produces semantically incorrect code.
os/aix
os/android
os/bridgeos
os/contiki
os/dragonfly
os/driverkit
os/emscripten
os/freebsd
os/fuchsia
os/haiku
os/hermit
os/hurd
os/illumos
os/ios
os/kfreebsd
os/linux
os/maccatalyst
os/macos
os/managarm
os/netbsd
os/ohos
os/openbsd
os/plan9
os/redox
os/rtems
os/serenity
os/solaris
os/tvos
os/uefi
os/visionos
os/wali
os/wasi
os/watchos
os/windows
os/zos
proposal

This issue suggests modifications. If it also has the "accepted" label then it is planned.
release notes

This issue or pull request should be mentioned in the release notes.
testing

This issue is related to testing the compiler, standard library, or other parts of Zig.
tier system

This issue tracks the support tier for a target.
zig cc

Zig as a drop-in C-family compiler.
zig fmt

The Zig source code formatter.
bounty

https://ziglang.org/news/announcing-donor-bounties
bug

Observed behavior contradicts documented or intended behavior.
contributor-friendly

This issue is limited in scope and/or knowledge of project internals.
downstream

An issue with a third-party project that uses this project.
enhancement

Solving this issue will likely involve adding new logic or components to the codebase.
infra

An issue related to project infrastructure, e.g. continuous integration.
optimization

A task to improve performance and/or resource usage.
question

No questions on the issue tracker; use a community space instead.
regression

A bug that did not occur in a previous version.
upstream

An issue with a third-party project that this project uses.
No labels
abi/f32
abi/ilp32
abi/n32
abi/sf
abi/x32
accepted
arch/1750a
arch/21k
arch/6502
arch/a29k
arch/aarch64
arch/alpha
arch/amdgcn
arch/arc
arch/arc32
arch/arc64
arch/arm
arch/avr
arch/avr32
arch/bfin
arch/bpf
arch/clipper
arch/colossus
arch/cr16
arch/cris
arch/csky
arch/dlx
arch/dsp16xx
arch/elxsi
arch/epiphany
arch/fr30
arch/frv
arch/h8300
arch/h8500
arch/hexagon
arch/hppa
arch/hppa64
arch/i370
arch/i860
arch/i960
arch/ia64
arch/ip2k
arch/kalimba
arch/kvx
arch/lanai
arch/lm32
arch/loongarch32
arch/loongarch64
arch/m32r
arch/m68k
arch/m88k
arch/maxq
arch/mcore
arch/metag
arch/microblaze
arch/mips
arch/mips64
arch/mmix
arch/mn10200
arch/mn10300
arch/moxie
arch/mrisc32
arch/msp430
arch/nds32
arch/nios2
arch/ns32k
arch/nvptx
arch/or1k
arch/pdp10
arch/pdp11
arch/pj
arch/powerpc
arch/powerpc64
arch/propeller
arch/riscv32
arch/riscv64
arch/rl78
arch/rx
arch/s390
arch/s390x
arch/sh
arch/sh64
arch/sparc
arch/sparc64
arch/spirv
arch/spu
arch/st200
arch/starcore
arch/tilegx
arch/tilepro
arch/tricore
arch/ts
arch/v850
arch/vax
arch/vc4
arch/ve
arch/wasm
arch/we32k
arch/x86
arch/x86_16
arch/x86_64
arch/xcore
arch/xgate
arch/xstormy16
arch/xtensa
autodoc
backend/c
backend/llvm
backend/self-hosted
binutils
breaking
build system
debug info
docs
error message
frontend
fuzzing
incremental
lib/c
lib/compiler-rt
lib/cxx
lib/std
lib/tsan
lib/ubsan-rt
lib/unwind
linking
miscompilation
os/aix
os/android
os/bridgeos
os/contiki
os/dragonfly
os/driverkit
os/emscripten
os/freebsd
os/fuchsia
os/haiku
os/hermit
os/hurd
os/illumos
os/ios
os/kfreebsd
os/linux
os/maccatalyst
os/macos
os/managarm
os/netbsd
os/ohos
os/openbsd
os/plan9
os/redox
os/rtems
os/serenity
os/solaris
os/tvos
os/uefi
os/visionos
os/wali
os/wasi
os/watchos
os/windows
os/zos
proposal
release notes
testing
tier system
zig cc
zig fmt
bounty
bug
contributor-friendly
downstream
enhancement
infra
optimization
question
regression
upstream
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
ziglang/zig!30788
Reference in a new issue
ziglang/zig
No description provided.
Delete branch "debug-file-leaks"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?