Coff linker enhancements, new linker testing framework, and COFF objdump implementation #35674

kcbanner commented 2026年06月08日 06:40:49 +02:00

This changeset enhances the Coff linker with the following features:

• Outputing objects (.obj) and archives (.lib)
• Outputing implibs alongside images
• Outputing the TLS and Exports data directories for images
• Consuming objects, archives, and import libraries as inputs
• Only links in objects from archives as required, ie. if they contain a symbol needed to satisfy a reference
• COMDAT rules (enough support for linking compiler_rt and libc, some COMDAT types are not supported yet)
• Supports linking against both -gnu and -msvc libc
• TLS support
• __dllimport support: Indirect calls / loads from the IAT directly
• Detects which entrypoint to choose based on exported symbols
• -gnu: Constructor / destructor support (ie. merge .ctor and .dtor, and set up the __CTOR_LIST__, __DTOR_LIST__ symbols)
• Support for several .drectve arguments (these are required to correctly link msvc libc):
  • /INCLUDE: Forcing a symbol to be referenced
  • /ALTERNATENAME: Adding symbol aliases
  • /MERGE: Section merging. This functionality is also used to direct certain sections into the right place (like .ctor / .dtor into .rdata)
  • /DEFAULTLIB: Adding new inputs
• Enhanced the --debug-link-snapshot output

Supporting changes:

• Linker crash reporting now can optionally dump a linker snapshot if --debug-link-snapshot is provided. This is not enabled by default as dumping snapshots for large binaries can take some time, and gets in the way if actually debugging the process.
• x86_64/Emit: Support __dllimported functions / data by codegening the indirect call / load to the __imp_ symbol directly.
• MappedFile: Added realign, shrink, and unit tests
• MappedFile: A few fixes to resizing, mainly around when sibling nodes don't have the same alignment. Linking libc adds a lot of input section, and they tend to vary in alignment, so I hit this case a lot.
• MappedFile: Added a safety lock. This is used in idle() as to prevent accidently introducing a node modification in an idle task, which would break deterministic output.
• meta: Add BareUnion (from multi_array_list)

Additionally, this PR also contains the new linker testing framework, and objdump implementation for COFF (see below).

I removed the legacy compiler_rt dynlib hack as it's no longer needed, we now support all the COMDAT features required to link compiler_rt statically.

See the added linker tests for more details, but in summary: the COFF linker generates working binaries for both gnu and msvc (including libc), linking statically against compiler_rt!

There are several remaining TODOs in Coff.zig, which I plan on addressing in future PRs. I didn't want to hold off on PRing this branch any longer, as some of these are non-trivial, and I haven't figured out how to address them yet.

New linker testing framework

I've added a new set of linker tests (which were removed in #32065), in a new form that aims to address the points outlined in that pr:

• less effort to create and maintain tests
• less CPU overhead - we should be able to add a lot of tests without
  adding a lot of CI time.
• more helpful failures. A failed linker test should provide the next
  steps a developer can take to understand why the test failed.
• a goal of porting over all of LLD and mold's test suites, or at least the good
  ones.

The new framework works as follows:

• test-link is the root step
• Test targets are defined explicitly in test.link_targets
• I've chosen to only run .Debug targets for now, to reduce the CI cost. Both llvm and self-hosted linkers are tested.
• Tests are a combination of comparing objdump snapshot output, actually running the artifacts, and checking for linker errors
• verifyObjdump is used to compare existing snapshots to zig objdump output for an artifact
• -Dlink-snapshot-update causes tests to run in a mode that outputs snapshots instead of checking against them

A typical workflow for adding a new test:

1. Add the test
2. zig-debug build test-link -Dtest-filter=my-test -Dlink-snapshot-update
   • This will output a .dmp file for all the snapshot combinations defined in any verifyObjdump calls
   • A single snapshot intentionally aliaes between many targets to reduce noise in the snapshot folder, so snapshot updates are made by whichever test runs first for that snapshot name. If differences between targets do exist, they will be revealed in step 4.
3. Inspect the snapshot output for correctness
4. zig-debug build test-link -Dtest-filter=my-test
   • This will now run all targets against the newly added snapshots
5. If there are now snapshot failures, that means different targets had different snapshot outputs. The output should be inspected to see if these results are indeed valid differences. If they are, then the scope parameter should be used to cause -Dlink-snapshot-update to output snapshot to different filenames, scoped on the diference.
6. Re-run -Dlink-snapshot-update to update the new set of snapshots

I've only added COFF targets for these tests, as they rely on objdump functionality which only exists for COFF so far.

I plan on enhancing this test framework further in futures PRs (alongside objdump, see below), to address the 4th point about porting over tests from other linkers. There are also lots of missing tests that still need to be added back.

COFF Objdump

This was a requirement for snapshot testing, as well as aiding in developing the linker itself

Supported features:

Usage: zig objdump [options] file
Options:
 -h, --help Print this help and exit
 --all-headers Alias for --file-headers --linker-member=2 --member-headers --section-headers --relocs --symbols
 --exports[=sort] Display exported symbols.
 In the case of COFF import libraries, displays the symbol list and import headers.
 Specify =sort to optionally sort the import headers by symbol name.
 --file-headers Display file-format specific headers
 --imports Display imported symbols
 --linker-member[=1|2|longnames] (Coff) Display contents of the specified archive linker member (default 2)
 --member-headers Display archive member headers
 --elements=[e1],[e2],-[e3],... Select which formatting elements are displayed. Intended for snapshot testing.
 file-type File type summary
 header-name Name that precedes a header block
 member-path Display full member paths. If removed, only basenames will be used.
 newlines Newlines between output sections
 table-header Table headers with column names
 all (default) All of the above
 --only-member=[name] Only consider archive members names that contain [name]. Can be specified multiple times.
 --only-section=[name] Only consider section names that contain [name]. Can be specified multiple times.
 --only-symbol=[name] Only consider symbol names that contain [name]. Can be specified multiple times.
 --redact=[kind] Redact the specified field kind. Intended for snapshot testing.
 rva Relative virtual addresses
 va Virtual addresses and file offsets
 ord Symbol ordinals / hints
 size Sizes and lengths
 all All of the above
 --relocs Display relocations
 -s, --snapshot Alias for --redact=all --elements=-all
 --section-headers Display section headers
 --strings Display string tables
 --symbols Display symbol tables
 --tls Display TLS information

Some example output:

❯ zig objdump mathtest-dync-exe-no-llvm.dll --all-headers
mathtest-dync-exe-no-llvm.dll: PE/COFF image
COFF Header:
 8664 machine (AMD64)
 7 number_of_sections
 6a08670a time_date_stamp
 0 pointer_to_symbol_table
 0 number_of_symbols
 f0 size_of_optional_header
 2022 flags
 | EXECUTABLE_IMAGE
 | LARGE_ADDRESS_AWARE
 | DLL
COFF Optional Header:
 20b magic (PE32+)
 14.00 linker_version
 14a400 size_of_code
 84e00 size_of_initialized_data
 0 size_of_uninitialized_data
 1000 address_of_entry_point ( 180001000)
 1000 base_of_code ( 180001000)
 180000000 image_base
 1000 section_alignment
 200 file_alignment
 6.00 operating_system_version
 1.00 image_version
 6.00 subsystem_version
 0 win32_version_value
 1d6000 size_of_image
 400 size_of_headers
 0 checksum
 2 subsystem (WINDOWS_GUI)
 160 dll_flags
 | HIGH_ENTROPY_VA
 | DYNAMIC_BASE
 | NX_COMPAT
 100000 size_of_stack_reserve
 1000 size_of_stack_commit
 100000 size_of_heap_reserve
 1000 size_of_heap_commit
 0 loader_flags
 10 number_of_rva_and_sizes
Data Directories:
 1b48d0 54 EXPORT
 1b4924 8c IMPORT
 0 0 RESOURCE
 1cc000 633c EXCEPTION
 0 0 SECURITY
 1d4000 1280 BASERELOC
 1bc000 1c DEBUG
 0 0 ARCHITECTURE
 0 0 GLOBALPTR
 1aebc8 28 TLS
 0 0 LOAD_CONFIG
 0 0 BOUND_IMPORT
 1b4c80 2d0 IAT
 0 0 DELAY_IMPORT
 0 0 COM_DESCRIPTOR
 0 0 RESERVED
Sections in 'mathtest-dync-exe-no-llvm.dll':
Num Name RVA Virt Size Data Size & Data & Relocs & Lines # Relocs # Lines Flags
 1 .text 1000 14a346 14a400 400 0 0 0 0 60000020 | CNT_CODE MEM_EXECUTE MEM_READ
 2 .rdata 14c000 6fe3c 70000 14a800 0 0 0 0 40000040 | CNT_INITIALIZED_DATA MEM_READ
 3 .buildid 1bc000 52 200 1ba800 0 0 0 0 40000040 | CNT_INITIALIZED_DATA MEM_READ
 4 .data 1bd000 e2a0 d200 1baa00 0 0 0 0 c0000040 | CNT_INITIALIZED_DATA MEM_READ MEM_WRITE
 5 .pdata 1cc000 633c 6400 1c7c00 0 0 0 0 40000040 | CNT_INITIALIZED_DATA MEM_READ
 6 .tls 1d3000 20 200 1ce000 0 0 0 0 c0000040 | CNT_INITIALIZED_DATA MEM_READ MEM_WRITE
 7 .reloc 1d4000 1280 1400 1ce200 0 0 0 0 42000040 | CNT_INITIALIZED_DATA MEM_DISCARDABLE MEM_READ
No symbol table found

If this output was used for a snapshot test that wanted to test the presence of a particular export:

❯ zig objdump mathtest-dync-exe-no-llvm.dll --exports --only-symbol=add --redact=rva --elements=-all
Export directory:
 0 flags
 0 time_date_stamp
 0.00 version
xxxxxxxxxxxxxxxx name_rva
 1 ordinal_base
 1 number_of_entries
 1 number_of_names
xxxxxxxxxxxxxxxx export_address_table_rva
xxxxxxxxxxxxxxxx name_pointer_table_rva
xxxxxxxxxxxxxxxx ordinal_table_rva
 1 0 xxxxxxxx | add

The redaction (--redact=) and element removal (--elements=) functionality is used to remove parts of the output that don't matter for the particular test, as to not cause spurious failures if say, an RVA for a symbol changes due to some unrelated change to the linker. -s is shorthand which enables all the redacts and removes all extra output elements, but particular tests may not use this if they do care about specific values.

This changeset enhances the Coff linker with the following features: - Outputing objects (.obj) and archives (.lib) - Outputing implibs alongside images - Outputing the TLS and Exports data directories for images - Consuming objects, archives, and import libraries as inputs - Only links in objects from archives as required, ie. if they contain a symbol needed to satisfy a reference - COMDAT rules (enough support for linking compiler_rt and libc, some COMDAT types are not supported yet) - Supports linking against both -gnu and -msvc libc - TLS support - `__dllimport` support: Indirect calls / loads from the IAT directly - Detects which entrypoint to choose based on exported symbols - `-gnu`: Constructor / destructor support (ie. merge .ctor and .dtor, and set up the `__CTOR_LIST__`, `__DTOR_LIST__` symbols) - Support for several `.drectve` arguments (these are required to correctly link `msvc` libc): - `/INCLUDE`: Forcing a symbol to be referenced - `/ALTERNATENAME`: Adding symbol aliases - `/MERGE`: Section merging. This functionality is also used to direct certain sections into the right place (like `.ctor` / `.dtor` into `.rdata`) - `/DEFAULTLIB`: Adding new inputs - Enhanced the `--debug-link-snapshot` output Supporting changes: - Linker crash reporting now can optionally dump a linker snapshot if `--debug-link-snapshot` is provided. This is not enabled by default as dumping snapshots for large binaries can take some time, and gets in the way if actually debugging the process. - x86_64/Emit: Support `__dllimport`ed functions / data by codegening the indirect call / load to the `__imp_` symbol directly. - MappedFile: Added realign, shrink, and unit tests - MappedFile: A few fixes to resizing, mainly around when sibling nodes don't have the same alignment. Linking libc adds a lot of input section, and they tend to vary in alignment, so I hit this case a lot. - MappedFile: Added a safety lock. This is used in `idle()` as to prevent accidently introducing a node modification in an idle task, which would break deterministic output. - meta: Add BareUnion (from `multi_array_list`) Additionally, this PR also contains the new linker testing framework, and `objdump` implementation for COFF (see below). I removed the legacy compiler_rt dynlib hack as it's no longer needed, we now support all the COMDAT features required to link compiler_rt statically. See the added linker tests for more details, but in summary: the COFF linker generates working binaries for both `gnu` and `msvc` (including libc), linking statically against `compiler_rt`! There are several remaining TODOs in Coff.zig, which I plan on addressing in future PRs. I didn't want to hold off on PRing this branch any longer, as some of these are non-trivial, and I haven't figured out how to address them yet. ### New linker testing framework I've added a new set of linker tests (which were removed in https://codeberg.org/ziglang/zig/pulls/32065), in a new form that aims to address the points outlined in that pr: > - less effort to create and maintain tests > - less CPU overhead - we should be able to add a lot of tests without adding a lot of CI time. > - more helpful failures. A failed linker test should provide the next steps a developer can take to understand why the test failed. > - a goal of porting over all of LLD and mold's test suites, or at least the good ones. The new framework works as follows: - `test-link` is the root step - Test targets are defined explicitly in `test.link_targets` - I've chosen to only run .Debug targets for now, to reduce the CI cost. Both llvm and self-hosted linkers are tested. - Tests are a combination of comparing objdump snapshot output, actually running the artifacts, and checking for linker errors - `verifyObjdump` is used to compare existing snapshots to `zig objdump` output for an artifact - `-Dlink-snapshot-update` causes tests to run in a mode that outputs snapshots instead of checking against them A typical workflow for adding a new test: 1. Add the test 2. zig-debug build test-link -Dtest-filter=my-test -Dlink-snapshot-update - This will output a .dmp file for all the snapshot combinations defined in any `verifyObjdump` calls - A single snapshot intentionally aliaes between many targets to reduce noise in the snapshot folder, so snapshot updates are made by whichever test runs first for that snapshot name. If differences between targets do exist, they will be revealed in step 4. 3. Inspect the snapshot output for correctness 4. zig-debug build test-link -Dtest-filter=my-test - This will now run all targets against the newly added snapshots 5. If there are now snapshot failures, that means different targets had different snapshot outputs. The output should be inspected to see if these results are indeed valid differences. If they are, then the `scope` parameter should be used to cause `-Dlink-snapshot-update` to output snapshot to different filenames, scoped on the diference. 6. Re-run `-Dlink-snapshot-update` to update the new set of snapshots I've only added COFF targets for these tests, as they rely on objdump functionality which only exists for COFF so far. I plan on enhancing this test framework further in futures PRs (alongside objdump, see below), to address the 4th point about porting over tests from other linkers. There are also lots of missing tests that still need to be added back. ### COFF Objdump This was a requirement for snapshot testing, as well as aiding in developing the linker itself Supported features: ``` Usage: zig objdump [options] file Options: -h, --help Print this help and exit --all-headers Alias for --file-headers --linker-member=2 --member-headers --section-headers --relocs --symbols --exports[=sort] Display exported symbols. In the case of COFF import libraries, displays the symbol list and import headers. Specify =sort to optionally sort the import headers by symbol name. --file-headers Display file-format specific headers --imports Display imported symbols --linker-member[=1|2|longnames] (Coff) Display contents of the specified archive linker member (default 2) --member-headers Display archive member headers --elements=[e1],[e2],-[e3],... Select which formatting elements are displayed. Intended for snapshot testing. file-type File type summary header-name Name that precedes a header block member-path Display full member paths. If removed, only basenames will be used. newlines Newlines between output sections table-header Table headers with column names all (default) All of the above --only-member=[name] Only consider archive members names that contain [name]. Can be specified multiple times. --only-section=[name] Only consider section names that contain [name]. Can be specified multiple times. --only-symbol=[name] Only consider symbol names that contain [name]. Can be specified multiple times. --redact=[kind] Redact the specified field kind. Intended for snapshot testing. rva Relative virtual addresses va Virtual addresses and file offsets ord Symbol ordinals / hints size Sizes and lengths all All of the above --relocs Display relocations -s, --snapshot Alias for --redact=all --elements=-all --section-headers Display section headers --strings Display string tables --symbols Display symbol tables --tls Display TLS information ``` Some example output: ``` ❯ zig objdump mathtest-dync-exe-no-llvm.dll --all-headers mathtest-dync-exe-no-llvm.dll: PE/COFF image COFF Header: 8664 machine (AMD64) 7 number_of_sections 6a08670a time_date_stamp 0 pointer_to_symbol_table 0 number_of_symbols f0 size_of_optional_header 2022 flags | EXECUTABLE_IMAGE | LARGE_ADDRESS_AWARE | DLL COFF Optional Header: 20b magic (PE32+) 14.00 linker_version 14a400 size_of_code 84e00 size_of_initialized_data 0 size_of_uninitialized_data 1000 address_of_entry_point ( 180001000) 1000 base_of_code ( 180001000) 180000000 image_base 1000 section_alignment 200 file_alignment 6.00 operating_system_version 1.00 image_version 6.00 subsystem_version 0 win32_version_value 1d6000 size_of_image 400 size_of_headers 0 checksum 2 subsystem (WINDOWS_GUI) 160 dll_flags | HIGH_ENTROPY_VA | DYNAMIC_BASE | NX_COMPAT 100000 size_of_stack_reserve 1000 size_of_stack_commit 100000 size_of_heap_reserve 1000 size_of_heap_commit 0 loader_flags 10 number_of_rva_and_sizes Data Directories: 1b48d0 54 EXPORT 1b4924 8c IMPORT 0 0 RESOURCE 1cc000 633c EXCEPTION 0 0 SECURITY 1d4000 1280 BASERELOC 1bc000 1c DEBUG 0 0 ARCHITECTURE 0 0 GLOBALPTR 1aebc8 28 TLS 0 0 LOAD_CONFIG 0 0 BOUND_IMPORT 1b4c80 2d0 IAT 0 0 DELAY_IMPORT 0 0 COM_DESCRIPTOR 0 0 RESERVED Sections in 'mathtest-dync-exe-no-llvm.dll': Num Name RVA Virt Size Data Size & Data & Relocs & Lines # Relocs # Lines Flags 1 .text 1000 14a346 14a400 400 0 0 0 0 60000020 | CNT_CODE MEM_EXECUTE MEM_READ 2 .rdata 14c000 6fe3c 70000 14a800 0 0 0 0 40000040 | CNT_INITIALIZED_DATA MEM_READ 3 .buildid 1bc000 52 200 1ba800 0 0 0 0 40000040 | CNT_INITIALIZED_DATA MEM_READ 4 .data 1bd000 e2a0 d200 1baa00 0 0 0 0 c0000040 | CNT_INITIALIZED_DATA MEM_READ MEM_WRITE 5 .pdata 1cc000 633c 6400 1c7c00 0 0 0 0 40000040 | CNT_INITIALIZED_DATA MEM_READ 6 .tls 1d3000 20 200 1ce000 0 0 0 0 c0000040 | CNT_INITIALIZED_DATA MEM_READ MEM_WRITE 7 .reloc 1d4000 1280 1400 1ce200 0 0 0 0 42000040 | CNT_INITIALIZED_DATA MEM_DISCARDABLE MEM_READ No symbol table found ``` If this output was used for a snapshot test that wanted to test the presence of a particular export: ``` ❯ zig objdump mathtest-dync-exe-no-llvm.dll --exports --only-symbol=add --redact=rva --elements=-all Export directory: 0 flags 0 time_date_stamp 0.00 version xxxxxxxxxxxxxxxx name_rva 1 ordinal_base 1 number_of_entries 1 number_of_names xxxxxxxxxxxxxxxx export_address_table_rva xxxxxxxxxxxxxxxx name_pointer_table_rva xxxxxxxxxxxxxxxx ordinal_table_rva 1 0 xxxxxxxx | add ``` The redaction (`--redact=`) and element removal (`--elements=`) functionality is used to remove parts of the output that don't matter for the particular test, as to not cause spurious failures if say, an RVA for a symbol changes due to some unrelated change to the linker. `-s` is shorthand which enables all the redacts and removes all extra output elements, but particular tests may not use this if they do care about specific values.

jacobly commented 2026年06月22日 02:14:15 +02:00

Can be rebased on master now, and don't forget to enable the tests from #35865.

Can be rebased on master now, and don't forget to enable the tests from #35865.

kcbanner commented 2026年06月22日 07:26:12 +02:00

Thanks! I need to fix up an issue where undefs from imported objects don't resolve properly if those globals were first seen with an explicit lib name. I'll rebase and test with your changes once I sort that out 👍

Thanks! I need to fix up an issue where undefs from imported objects don't resolve properly if those globals were first seen with an explicit lib name. I'll rebase and test with your changes once I sort that out 👍