ziglang/zig
149
2.9k
Fork
You've already forked zig
279

introduce std.Io.File.MemoryMap API #30840

Merged
andrewrk merged 19 commits from mmap into master 2026年01月16日 03:59:33 +01:00

Progress towards #30150.

By defining the pointer contents to only be synchronized after explicit sync points, makes it legal to have a fallback implementation based on file operations while still supporting a handful of use cases for memory mapping.

Furthermore, it makes it legal for evented I/O implementations to use evented file I/O for the sync points rather than memory mapping.

Real world use case: src/link/MappedFile.zig

It's marked a breaking change because I constrained the positional file reading and writing error sets. Also on WASI, you now get error.IsDir correctly instead of error.NotOpenForReading.

Followup Tasks

  • update src/link/MappedFile.zig in the compiler to use this
  • audit the remaining calls to posix.mmap
  • handle populate=true on windows using NtSetInformationVirtualMemory with the info class as VmPrefetchInformation
  • investigate use of NtExtendSection
Progress towards #30150. By defining the pointer contents to only be synchronized after explicit sync points, makes it legal to have a fallback implementation based on file operations while still supporting a handful of use cases for memory mapping. Furthermore, it makes it legal for evented I/O implementations to use evented file I/O for the sync points rather than memory mapping. Real world use case: src/link/MappedFile.zig It's marked a breaking change because I constrained the positional file reading and writing error sets. Also on WASI, you now get `error.IsDir` correctly instead of `error.NotOpenForReading`. ## Followup Tasks * update src/link/MappedFile.zig in the compiler to use this * audit the remaining calls to posix.mmap * handle populate=true on windows using NtSetInformationVirtualMemory with the info class as VmPrefetchInformation * investigate use of NtExtendSection
by defining the pointer contents to only be synchronized after explicit
sync points, makes it legal to have a fallback implementation based on
file operations while still supporting a handful of use cases for memory
mapping.
furthermore, it makes it legal for evented I/O implementations to use
evented file I/O for the sync points rather than memory mapping.
not yet done:
- implement checking the length when options.len is null
- some windows impl work
- some wasi impl work
- unit tests
- integration with compiler
- change offset to u64
- make len non-optional
- make write take a file_size parameter
- std.Io.Threaded: introduce disable_memory_mapping flag to force it to
 take the fallback path.
Additionally:
- introduce BlockSize to File.Stat. On Windows, based on cached call to
 NtQuerySystemInformation. On unsupported OS's, set to 1.
- support File.NLink on Windows. this was available the whole time, we
 just didn't see the field at first.
- remove EBADF / INVALID_HANDLE from reading/writing file error sets
- remove file_size parameter from MemoryMap.write
- remove requirement for mapping length to be aligned
- align allocated fallback memory
- add unit test for std.Io.Threaded.disable_memory_mapping = true
- add unit test for MemoryMap.setLength
why were we testing for shitty behavior?
link: add some errors to the Dwarf.UpdateError mega error set
Some checks failed
ci / aarch64-linux-debug (pull_request) Has been cancelled
ci / aarch64-linux-release (pull_request) Has been cancelled
ci / aarch64-macos-debug (pull_request) Has been cancelled
ci / aarch64-macos-release (pull_request) Has been cancelled
ci / loongarch64-linux-debug (pull_request) Has been cancelled
ci / loongarch64-linux-release (pull_request) Has been cancelled
ci / powerpc64le-linux-release (pull_request) Has been cancelled
ci / s390x-linux-debug (pull_request) Has been cancelled
ci / s390x-linux-release (pull_request) Has been cancelled
ci / x86_64-freebsd-debug (pull_request) Has been cancelled
ci / x86_64-freebsd-release (pull_request) Has been cancelled
ci / x86_64-linux-debug (pull_request) Has been cancelled
ci / x86_64-linux-debug-llvm (pull_request) Has been cancelled
ci / x86_64-linux-release (pull_request) Has been cancelled
ci / x86_64-openbsd-debug (pull_request) Has been cancelled
ci / x86_64-openbsd-release (pull_request) Has been cancelled
ci / x86_64-windows-release (pull_request) Has been cancelled
ci / x86_64-windows-debug (pull_request) Has been cancelled
ci / powerpc64le-linux-debug (pull_request) Has been cancelled
eb629eab64
@ -0,0 +55,4 @@
///Prefaultthepages.Ifthisoptionisunsupported,itissilently
///ignored.AsidefromcustomIoimplementations,thisoptionisonly
///supportedonLinux.
populate:bool=true,
Contributor
Copy link

im not sure what the ntdll version is, but the kernel32 function PrefetchVirtualMemory can be used to prefault/prepopulate the pages

im not sure what the ntdll version is, but the kernel32 function PrefetchVirtualMemory can be used to prefault/prepopulate the pages

NtSetInformationVirtualMemory with the info class as VmPrefetchInformation

[`NtSetInformationVirtualMemory`](https://ntdoc.m417z.com/ntsetinformationvirtualmemory) with the info class as `VmPrefetchInformation`
Author
Owner
Copy link

would welcome that as a follow-up contribution

would welcome that as a follow-up contribution
tracked by #20747
std: fix handling of EBADF error code
Some checks failed
ci / aarch64-macos-release (pull_request) Successful in 30m22s
ci / x86_64-freebsd-release (pull_request) Successful in 34m19s
ci / x86_64-windows-release (pull_request) Failing after 35m30s
ci / x86_64-freebsd-debug (pull_request) Successful in 39m53s
ci / x86_64-openbsd-release (pull_request) Successful in 49m21s
ci / x86_64-openbsd-debug (pull_request) Successful in 53m59s
ci / x86_64-linux-debug (pull_request) Successful in 53m57s
ci / x86_64-windows-debug (pull_request) Failing after 56m8s
ci / aarch64-macos-debug (pull_request) Successful in 57m23s
ci / x86_64-linux-release (pull_request) Failing after 58m49s
ci / aarch64-linux-release (pull_request) Successful in 1h36m41s
ci / x86_64-linux-debug-llvm (pull_request) Failing after 2h13m10s
ci / aarch64-linux-debug (pull_request) Successful in 2h22m11s
ci / powerpc64le-linux-release (pull_request) Successful in 1h37m14s
ci / powerpc64le-linux-debug (pull_request) Successful in 3h57m43s
ci / s390x-linux-release (pull_request) Successful in 2h5m19s
ci / s390x-linux-debug (pull_request) Successful in 3h0m30s
ci / loongarch64-linux-release (pull_request) Successful in 2h33m17s
ci / loongarch64-linux-debug (pull_request) Successful in 2h58m27s
6c1cd9683b 
In the context of read/write it's ambiguous, means file was opened
without read/write respectively.
@ -16110,0 +16472,4 @@
fnfileMemoryMapWrite(userdata:?*anyopaque,mm:*File.MemoryMap)File.WritePositionalError!void{
constt:*Threaded=@ptrCast(@alignCast(userdata));
_=t;
if(mm.section!=null)return;
Contributor
Copy link

can use NtFlushVirtualMemory here on windows

can use NtFlushVirtualMemory here on windows
Author
Owner
Copy link

why?

why?
Contributor
Copy link

it triggers a flush to the file, which seems to be roughly what this function is doing (its just that in theory the os is doing it automatically normally when mapped as a section). the main benefit to using it is for safety in case of a crash/failure, making sure memory changes get flushed to disk before the memory gets unloaded if the process were to die

it triggers a flush to the file, which seems to be roughly what this function is doing (its just that in theory the os is doing it automatically normally when mapped as a section). the main benefit to using it is for safety in case of a crash/failure, making sure memory changes get flushed to disk before the memory gets unloaded if the process were to die
Contributor
Copy link

(the kernel32 equivalent is FlushViewOfFile)

(the kernel32 equivalent is FlushViewOfFile)
andrewrk force-pushed mmap from 6c1cd9683b
Some checks failed
ci / aarch64-macos-release (pull_request) Successful in 30m22s
ci / x86_64-freebsd-release (pull_request) Successful in 34m19s
ci / x86_64-windows-release (pull_request) Failing after 35m30s
ci / x86_64-freebsd-debug (pull_request) Successful in 39m53s
ci / x86_64-openbsd-release (pull_request) Successful in 49m21s
ci / x86_64-openbsd-debug (pull_request) Successful in 53m59s
ci / x86_64-linux-debug (pull_request) Successful in 53m57s
ci / x86_64-windows-debug (pull_request) Failing after 56m8s
ci / aarch64-macos-debug (pull_request) Successful in 57m23s
ci / x86_64-linux-release (pull_request) Failing after 58m49s
ci / aarch64-linux-release (pull_request) Successful in 1h36m41s
ci / x86_64-linux-debug-llvm (pull_request) Failing after 2h13m10s
ci / aarch64-linux-debug (pull_request) Successful in 2h22m11s
ci / powerpc64le-linux-release (pull_request) Successful in 1h37m14s
ci / powerpc64le-linux-debug (pull_request) Successful in 3h57m43s
ci / s390x-linux-release (pull_request) Successful in 2h5m19s
ci / s390x-linux-debug (pull_request) Successful in 3h0m30s
ci / loongarch64-linux-release (pull_request) Successful in 2h33m17s
ci / loongarch64-linux-debug (pull_request) Successful in 2h58m27s
to 59d2ba81ab
Some checks failed
ci / x86_64-openbsd-release (pull_request) Failing after 51m8s
ci / x86_64-freebsd-debug (pull_request) Failing after 45m8s
ci / x86_64-freebsd-release (pull_request) Failing after 32m38s
ci / x86_64-windows-release (pull_request) Failing after 39m57s
ci / aarch64-macos-release (pull_request) Failing after 1h2m27s
ci / x86_64-windows-debug (pull_request) Failing after 48m42s
ci / x86_64-openbsd-debug (pull_request) Failing after 1h11m18s
ci / x86_64-linux-debug (pull_request) Successful in 1h32m2s
ci / powerpc64le-linux-release (pull_request) Successful in 1h34m5s
ci / aarch64-macos-debug (pull_request) Failing after 1h42m1s
ci / aarch64-linux-release (pull_request) Successful in 1h51m28s
ci / x86_64-linux-release (pull_request) Failing after 1h54m50s
ci / aarch64-linux-debug (pull_request) Successful in 2h16m39s
ci / x86_64-linux-debug-llvm (pull_request) Failing after 2h53m46s
ci / loongarch64-linux-debug (pull_request) Has been cancelled
ci / loongarch64-linux-release (pull_request) Has been cancelled
ci / s390x-linux-release (pull_request) Has been cancelled
ci / powerpc64le-linux-debug (pull_request) Has been cancelled
ci / s390x-linux-debug (pull_request) Has been cancelled
2026年01月15日 19:45:15 +01:00 Compare
andrewrk force-pushed mmap from 59d2ba81ab
Some checks failed
ci / x86_64-openbsd-release (pull_request) Failing after 51m8s
ci / x86_64-freebsd-debug (pull_request) Failing after 45m8s
ci / x86_64-freebsd-release (pull_request) Failing after 32m38s
ci / x86_64-windows-release (pull_request) Failing after 39m57s
ci / aarch64-macos-release (pull_request) Failing after 1h2m27s
ci / x86_64-windows-debug (pull_request) Failing after 48m42s
ci / x86_64-openbsd-debug (pull_request) Failing after 1h11m18s
ci / x86_64-linux-debug (pull_request) Successful in 1h32m2s
ci / powerpc64le-linux-release (pull_request) Successful in 1h34m5s
ci / aarch64-macos-debug (pull_request) Failing after 1h42m1s
ci / aarch64-linux-release (pull_request) Successful in 1h51m28s
ci / x86_64-linux-release (pull_request) Failing after 1h54m50s
ci / aarch64-linux-debug (pull_request) Successful in 2h16m39s
ci / x86_64-linux-debug-llvm (pull_request) Failing after 2h53m46s
ci / loongarch64-linux-debug (pull_request) Has been cancelled
ci / loongarch64-linux-release (pull_request) Has been cancelled
ci / s390x-linux-release (pull_request) Has been cancelled
ci / powerpc64le-linux-debug (pull_request) Has been cancelled
ci / s390x-linux-debug (pull_request) Has been cancelled
to 7b21fd7244
Some checks failed
ci / aarch64-macos-release (pull_request) Successful in 38m7s
ci / x86_64-freebsd-release (pull_request) Successful in 43m21s
ci / x86_64-freebsd-debug (pull_request) Successful in 49m3s
ci / x86_64-windows-debug (pull_request) Successful in 53m47s
ci / x86_64-windows-release (pull_request) Successful in 43m22s
ci / x86_64-openbsd-release (pull_request) Successful in 55m38s
ci / aarch64-macos-debug (pull_request) Successful in 1h13m20s
ci / x86_64-openbsd-debug (pull_request) Successful in 1h11m15s
ci / aarch64-linux-release (pull_request) Successful in 1h27m34s
ci / powerpc64le-linux-release (pull_request) Successful in 1h35m53s
ci / x86_64-linux-debug (pull_request) Successful in 2h8m57s
ci / aarch64-linux-debug (pull_request) Successful in 2h38m41s
ci / x86_64-linux-debug-llvm (pull_request) Successful in 2h47m15s
ci / x86_64-linux-release (pull_request) Successful in 2h51m39s
ci / powerpc64le-linux-debug (pull_request) Successful in 4h1m52s
ci / loongarch64-linux-debug (pull_request) Has been cancelled
ci / loongarch64-linux-release (pull_request) Has been cancelled
ci / s390x-linux-debug (pull_request) Has been cancelled
ci / s390x-linux-release (pull_request) Has been cancelled
2026年01月15日 23:18:29 +01:00 Compare
Sign in to join this conversation.
No reviewers
Labels
Clear labels
abi/f32
abi/ilp32
abi/n32
abi/sf
abi/x32
accepted

This proposal is planned.
arch/1750a
arch/21k
arch/6502
arch/a29k
arch/aarch64
arch/alpha
arch/amdgcn
arch/arc
arch/arc32
arch/arc64
arch/arm
arch/avr
arch/avr32
arch/bfin
arch/bpf
arch/clipper
arch/colossus
arch/cr16
arch/cris
arch/csky
arch/dlx
arch/dsp16xx
arch/elxsi
arch/epiphany
arch/fr30
arch/frv
arch/h8300
arch/h8500
arch/hexagon
arch/hppa
arch/hppa64
arch/i370
arch/i860
arch/i960
arch/ia64
arch/ip2k
arch/kalimba
arch/kvx
arch/lanai
arch/lm32
arch/loongarch32
arch/loongarch64
arch/m32r
arch/m68k
arch/m88k
arch/maxq
arch/mcore
arch/metag
arch/microblaze
arch/mips
arch/mips64
arch/mmix
arch/mn10200
arch/mn10300
arch/moxie
arch/mrisc32
arch/msp430
arch/nds32
arch/nios2
arch/ns32k
arch/nvptx
arch/or1k
arch/pdp10
arch/pdp11
arch/pj
arch/powerpc
arch/powerpc64
arch/propeller
arch/riscv32
arch/riscv64
arch/rl78
arch/rx
arch/s390
arch/s390x
arch/sh
arch/sh64
arch/sparc
arch/sparc64
arch/spirv
arch/spu
arch/st200
arch/starcore
arch/tilegx
arch/tilepro
arch/tricore
arch/ts
arch/v850
arch/vax
arch/vc4
arch/ve
arch/wasm
arch/we32k
arch/x86
arch/x86_16
arch/x86_64
arch/xcore
arch/xgate
arch/xstormy16
arch/xtensa
autodoc

The web application for interactive documentation and generation of its assets.
backend/c

The C backend outputs C source code.
backend/llvm

The LLVM backend outputs an LLVM bitcode module.
backend/self-hosted

The self-hosted backends produce machine code directly.
binutils

Zig's included binary utilities: zig ar, zig dlltool, zig lib, zig ranlib, zig objcopy, and zig rc.
breaking

Implementing this issue could cause existing code to no longer compile or have different behavior.
build system

The Zig build system - zig build, std.Build, the build runner, and package management.
debug info

An issue related to debug information (e.g. DWARF) produced by the Zig compiler.
docs

An issue with documentation, e.g. the language reference or standard library doc comments.
error message

This issue points out an error message that is unhelpful and should be improved.
frontend

Tokenization, parsing, AstGen, ZonGen, Sema, Legalize, and Liveness.
fuzzing

An issue related to Zig's integrated fuzz testing.
incremental

Reuse of internal compiler state for faster compilation.
lib/c

This issue relates to Zig's libc implementation and/or vendored libcs.
lib/compiler-rt

This issue relates to Zig's compiler-rt library.
lib/cxx

This issue relates to Zig's vendored libc++ and/or libc++abi.
lib/std

This issue relates to Zig's standard library.
lib/tsan

This issue relates to Zig's vendored libtsan.
lib/ubsan-rt

This issue relates to Zig's ubsan-rt library.
lib/unwind

This issue relates to Zig's vendored libunwind.
linking

Zig's integrated object file and incremental linker.
miscompilation

The compiler reports success but produces semantically incorrect code.
os/aix
os/android
os/bridgeos
os/contiki
os/dragonfly
os/driverkit
os/emscripten
os/freebsd
os/fuchsia
os/haiku
os/hermit
os/hurd
os/illumos
os/ios
os/kfreebsd
os/linux
os/maccatalyst
os/macos
os/managarm
os/netbsd
os/ohos
os/openbsd
os/plan9
os/redox
os/rtems
os/serenity
os/solaris
os/tvos
os/uefi
os/visionos
os/wali
os/wasi
os/watchos
os/windows
os/zos
proposal

This issue suggests modifications. If it also has the "accepted" label then it is planned.
release notes

This issue or pull request should be mentioned in the release notes.
testing

This issue is related to testing the compiler, standard library, or other parts of Zig.
tier system

This issue tracks the support tier for a target.
zig cc

Zig as a drop-in C-family compiler.
zig fmt

The Zig source code formatter.
bounty

https://ziglang.org/news/announcing-donor-bounties
bug

Observed behavior contradicts documented or intended behavior.
contributor-friendly

This issue is limited in scope and/or knowledge of project internals.
downstream

An issue with a third-party project that uses this project.
enhancement

Solving this issue will likely involve adding new logic or components to the codebase.
infra

An issue related to project infrastructure, e.g. continuous integration.
optimization

A task to improve performance and/or resource usage.
question

No questions on the issue tracker; use a community space instead.
regression

A bug that did not occur in a previous version.
upstream

An issue with a third-party project that this project uses.
No labels
abi/f32
abi/ilp32
abi/n32
abi/sf
abi/x32
accepted
arch/1750a
arch/21k
arch/6502
arch/a29k
arch/aarch64
arch/alpha
arch/amdgcn
arch/arc
arch/arc32
arch/arc64
arch/arm
arch/avr
arch/avr32
arch/bfin
arch/bpf
arch/clipper
arch/colossus
arch/cr16
arch/cris
arch/csky
arch/dlx
arch/dsp16xx
arch/elxsi
arch/epiphany
arch/fr30
arch/frv
arch/h8300
arch/h8500
arch/hexagon
arch/hppa
arch/hppa64
arch/i370
arch/i860
arch/i960
arch/ia64
arch/ip2k
arch/kalimba
arch/kvx
arch/lanai
arch/lm32
arch/loongarch32
arch/loongarch64
arch/m32r
arch/m68k
arch/m88k
arch/maxq
arch/mcore
arch/metag
arch/microblaze
arch/mips
arch/mips64
arch/mmix
arch/mn10200
arch/mn10300
arch/moxie
arch/mrisc32
arch/msp430
arch/nds32
arch/nios2
arch/ns32k
arch/nvptx
arch/or1k
arch/pdp10
arch/pdp11
arch/pj
arch/powerpc
arch/powerpc64
arch/propeller
arch/riscv32
arch/riscv64
arch/rl78
arch/rx
arch/s390
arch/s390x
arch/sh
arch/sh64
arch/sparc
arch/sparc64
arch/spirv
arch/spu
arch/st200
arch/starcore
arch/tilegx
arch/tilepro
arch/tricore
arch/ts
arch/v850
arch/vax
arch/vc4
arch/ve
arch/wasm
arch/we32k
arch/x86
arch/x86_16
arch/x86_64
arch/xcore
arch/xgate
arch/xstormy16
arch/xtensa
autodoc
backend/c
backend/llvm
backend/self-hosted
binutils
breaking
build system
debug info
docs
error message
frontend
fuzzing
incremental
lib/c
lib/compiler-rt
lib/cxx
lib/std
lib/tsan
lib/ubsan-rt
lib/unwind
linking
miscompilation
os/aix
os/android
os/bridgeos
os/contiki
os/dragonfly
os/driverkit
os/emscripten
os/freebsd
os/fuchsia
os/haiku
os/hermit
os/hurd
os/illumos
os/ios
os/kfreebsd
os/linux
os/maccatalyst
os/macos
os/managarm
os/netbsd
os/ohos
os/openbsd
os/plan9
os/redox
os/rtems
os/serenity
os/solaris
os/tvos
os/uefi
os/visionos
os/wali
os/wasi
os/watchos
os/windows
os/zos
proposal
release notes
testing
tier system
zig cc
zig fmt
bounty
bug
contributor-friendly
downstream
enhancement
infra
optimization
question
regression
upstream
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
ziglang/zig!30840
Reference in a new issue
ziglang/zig
No description provided.
Delete branch "mmap"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?