Using reverse engineering to create an open-source alternative to the CUDA Driver API

Directory Structure

• src/ - Library source code (cubin/fatbin parsers, driver interface)
• include/ - Public C API headers
• tests/ - Test files and sample binaries
  • behavior/ - Runnable behavior/integration tests
  • cubin/ - Cubin test binaries
  • fatbin/ - Fatbin test binaries
• tools/ - Helper scripts (Python utilities, GDB scripts)
• nix/ - Nix package derivations

System Requirements

• Linux 5.10+ or FreeBSD 14.0+ operating system.
• Official NVIDIA proprietary or open-source device drivers (not Nouveau), may already be pre-installed as part of the Linux distribution.
• NVIDIA GPU supporting open-source device drivers (Turing microarchitecture and newer).

Software Dependencies for Building

• (Optional) Nix package manager for installing other packages and development environment
• Zig version 0.16.0-dev.2860+9c5460316
• Python 3
• (Optional) GDB for using GDB scripts
• (Optional) QEMU for testing on different CPU architectures

Procedure

1. Clone the Project

git clone https://codeberg.org/tondik/recu
cd recu

2. Obtaining Dependencies

2.1. With Nix Package Manager (Recommended)

Here you can find installation instructions for Nix.

The project uses a Nix flake that defines a development environment with all necessary dependencies.

To enter the development environment, in the project root directory enter:

nix develop

2.2. Without Nix Package Manager

For most dependencies, you can use your preferred package manager, but Zig cannot be installed using a standard package manager because the project requires specifically version 0.16.0-dev.2860+9c5460316. There are several options:

1. Using zvm - Zig version manager.
2. Building from source, commit 9c5460316.
3. Downloading a precompiled binary (Zig is statically linked, you don't have to worry about wrong glibc version issues and similar). Files are signed using minisign with the public key found here.

3. Running Sample Examples

Sample examples are located in the tests/behavior directory within the project and also serve to test the correctness of the implementation.

To run a specific example, in the project root directory enter:

cd tests/behavior/<name>
zig build run

4. Building the Static Library and C API

For interoperability with other languages, a C API with header file include/recu.h is available. By building into a static library, the project can be used from any programming language supporting C ABI function calls.

In the project root directory enter:

nix build .

or

zig build

The compilation result will be in the zig-out/lib directory.

5. Additional Parameters for zig build

• -Doptimize - Optimization level (Debug, ReleaseSafe, ReleaseFast, ReleaseSmall).
• -Dllvm - Use LLVM backend. Enabled by default for non-debug builds.
• -Dlld - Use LLD linker. Enabled by default with -Dllvm.
• -Dstrip - Remove debug symbols. Enabled by default for non-debug builds.
• -Dtest-filters - Filter for tests.
• -Dtest-foreign - Run tests on foreign architectures using QEMU.
• --fuzz - Run fuzz testing.
• --help - Display help with all available parameters, project-specific parameters have the -D prefix.

6. Generating Documentation

Documentation is automatically generated from the source code.

In the project root directory enter:

zig build docs

The result will be in the zig-out/docs directory.

7. Helper Scripts and GDB Scripts

Helper scripts are located in the tools/ directory, for example for decrypting data from a binary file using the nvdisasm tool. Scripts have execution permissions set and a shebang using nix-shell.

GDB scripts are run using:

gdb -x <script> -q <monitored binary file>

Update

nix flake update
nix develop .full
zig fetch --save git+https://codeberg.org/ziglang/translate-c
zig fetch --save=open_gpu_kernel_modules "git+https://github.com/NVIDIA/open-gpu-kernel-modules"
zig fetch --save=open_gpu_doc "git+https://github.com/NVIDIA/open-gpu-doc"
zig fetch --save=lz4 "git+https://github.com/lz4/lz4#v1.10.0"
zig fetch --save=zstd "git+https://github.com/facebook/zstd#v1.5.7"
zig fetch --save=tracy "git+https://github.com/wolfpld/tracy.git#v0.13.1"
zon2nix --nix=build.zig.zon.nix build.zig.zon

License

This project is distributed under the terms of LGPL-3.0-or-later. See the LICENSE file for details.