diff --git a/.gitignore b/.gitignore index 15ad2465251..270795db918 100644 --- a/.gitignore +++ b/.gitignore @@ -32,4 +32,12 @@ lcov.info # Docker volumes and debug logs .postgres -logfile \ No newline at end of file +logfile + +# Nix related files +.direnv +.envrc +.data + +# Local claude settings +.claude/settings.local.json diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000000..4132af3a750 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,251 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Project Overview + +Graph Node is a Rust-based decentralized blockchain indexing protocol that enables efficient querying of blockchain data through GraphQL. It's the core component of The Graph protocol, written as a Cargo workspace with multiple crates organized by functionality. + +## Essential Development Commands + +### Testing Workflow + +⚠️ **Only run integration tests when explicitly requested or when changes require full system testing** + +Use unit tests for regular development and only run integration tests when: + +- Explicitly asked to do so +- Making changes to integration/end-to-end functionality +- Debugging issues that require full system testing +- Preparing releases or major changes + +### Unit Tests + +Unit tests are inlined with source code. + +**Prerequisites:** +1. PostgreSQL running on localhost:5432 (with initialised `graph-test` database) +2. IPFS running on localhost:5001 +3. Yarn (v1) +4. Foundry (for smart contract compilation) +5. Environment variable `THEGRAPH_STORE_POSTGRES_DIESEL_URL` set + +The environment dependencies and environment setup are operated by the human. + +**Running Unit Tests:** +```bash +# REQUIRED: Export database connection string before running unit tests (do this **once** at the beginning of a testing session) +export THEGRAPH_STORE_POSTGRES_DIESEL_URL="postgresql://graph:graph@127.0.0.1:5432/graph-test" + +# Run unit tests (excluding integration tests) +cargo test --workspace --exclude graph-tests + +# Run specific tests +cargo test -p graph data_source::common::tests +cargo test +``` + +### Runner Tests (Integration Tests) + +**Prerequisites:** +1. PostgreSQL running on localhost:5432 (with initialised `graph-test` database) +2. IPFS running on localhost:5001 +3. Yarn (v1) +4. Foundry (for smart contract compilation) +5. Environment variable `THEGRAPH_STORE_POSTGRES_DIESEL_URL` set + +**Running Runner Tests:** +```bash +# REQUIRED: Export database connection string before running unit tests (do this **once** at the beginning of a testing session) +export THEGRAPH_STORE_POSTGRES_DIESEL_URL="postgresql://graph:graph@127.0.0.1:5432/graph-test" + +# Run all runner tests +cargo test -p graph-tests --test runner_tests -- --nocapture + +# Run specific runner tests +cargo test -p graph-tests --test runner_tests test_name -- --nocapture +``` + +**Important Notes:** +- Runner tests take moderate time (10-20 seconds) +- Tests automatically reset the database between runs +- Some tests can pass without IPFS, but tests involving file data sources or substreams require it + +### Integration Tests + +**Prerequisites:** +1. PostgreSQL running on localhost:3011 (with initialised `graph-node` database) +2. IPFS running on localhost:3001 +3. Anvil running on localhost:3021 +4. Yarn (v1) +5. Foundry (for smart contract compilation) + +The environment dependencies and environment setup are operated by the human. + +**Running Integration Tests:** +```bash +# Run all integration tests +cargo test -p graph-tests --test integration_tests -- --nocapture + +# Run a specific integration test case (e.g., "grafted" test case) +TEST_CASE=grafted cargo test -p graph-tests --test integration_tests -- --nocapture +``` + +**Important Notes:** +- Integration tests take significant time (several minutes) +- Tests automatically reset the database between runs +- Logs are written to `tests/integration-tests/graph-node.log` + +### Code Quality +```bash +# 🚨 MANDATORY: Format all code IMMEDIATELY after any .rs file edit +cargo fmt --all + +# 🚨 MANDATORY: Check code for warnings and errors - MUST have zero warnings +cargo check +``` + +🚨 **CRITICAL REQUIREMENTS for ANY implementation**: +- **🚨 MANDATORY**: `cargo fmt --all` MUST be run before any commit +- **🚨 MANDATORY**: `cargo check` MUST show zero warnings before any commit +- **🚨 MANDATORY**: The unit test suite MUST pass before any commit + +Forgetting any of these means you failed to follow instructions. Before any commit or PR, ALL of the above MUST be satisfied! No exceptions! + +## High-Level Architecture + +### Core Components +- **`graph/`**: Core abstractions, traits, and shared types +- **`node/`**: Main executable and CLI (graphman) +- **`chain/`**: Blockchain-specific adapters (ethereum, near, substreams) +- **`runtime/`**: WebAssembly runtime for subgraph execution +- **`store/`**: PostgreSQL-based storage layer +- **`graphql/`**: GraphQL query execution engine +- **`server/`**: HTTP/WebSocket APIs + +### Data Flow +``` +Blockchain β†’ Chain Adapter β†’ Block Stream β†’ Trigger Processing β†’ Runtime β†’ Store β†’ GraphQL API +``` + +1. **Chain Adapters** connect to blockchain nodes and convert data to standardized formats +2. **Block Streams** provide event-driven streaming of blockchain blocks +3. **Trigger Processing** matches blockchain events to subgraph handlers +4. **Runtime** executes subgraph code in WebAssembly sandbox +5. **Store** persists entities with block-level granularity +6. **GraphQL** processes queries and returns results + +### Key Abstractions +- **`Blockchain`** trait: Core blockchain interface +- **`Store`** trait: Storage abstraction with read/write variants +- **`RuntimeHost`**: WASM execution environment +- **`TriggerData`**: Standardized blockchain events +- **`EventConsumer`/`EventProducer`**: Component communication + +### Architecture Patterns +- **Event-driven**: Components communicate through async streams and channels +- **Trait-based**: Extensive use of traits for abstraction and modularity +- **Async/await**: Tokio-based async runtime throughout +- **Multi-shard**: Database sharding for scalability +- **Sandboxed execution**: WASM runtime with gas metering + +## Development Guidelines + +### Commit Convention +Use format: `{crate-name}: {description}` +- Single crate: `store: Support 'Or' filters` +- Multiple crates: `core, graphql: Add event source to store` +- All crates: `all: {description}` + +### Git Workflow +- Rebase on master (don't merge master into feature branch) +- Keep commits logical and atomic +- Squash commits to clean up history before merging + +## Crate Structure + +### Core Crates +- **`graph`**: Shared types, traits, and utilities +- **`node`**: Main binary and component wiring +- **`core`**: Business logic and subgraph management + +### Blockchain Integration +- **`chain/ethereum`**: Ethereum chain support +- **`chain/near`**: NEAR protocol support +- **`chain/substreams`**: Substreams data source support + +### Infrastructure +- **`store/postgres`**: PostgreSQL storage implementation +- **`runtime/wasm`**: WebAssembly runtime and host functions +- **`graphql`**: Query processing and execution +- **`server/`**: HTTP/WebSocket servers + +### Key Dependencies +- **`diesel`**: PostgreSQL ORM +- **`tokio`**: Async runtime +- **`tonic`**: gRPC framework +- **`wasmtime`**: WebAssembly runtime +- **`web3`**: Ethereum interaction + +## Test Environment Requirements + +### Process Compose Setup (Recommended) + +The repository includes a process-compose-flake setup that provides native, declarative service management. + +Currently, the human is required to operate the service dependencies as illustrated below. + +**Unit Tests:** +```bash +# Human: Start PostgreSQL + IPFS for unit tests in a separate terminal +# PostgreSQL: localhost:5432, IPFS: localhost:5001 +nix run .#unit + +# Claude: Export the database connection string before running unit tests +export THEGRAPH_STORE_POSTGRES_DIESEL_URL="postgresql://graph:graph@127.0.0.1:5432/graph-test" + +# Claude: Run unit tests +cargo test --workspace --exclude graph-tests +``` + +**Runner Tests:** +```bash +# Human: Start PostgreSQL + IPFS for runner tests in a separate terminal +# PostgreSQL: localhost:5432, IPFS: localhost:5001 +nix run .#unit # NOTE: Runner tests are using the same nix services stack as the unit test + +# Claude: Export the database connection string before running runner tests +export THEGRAPH_STORE_POSTGRES_DIESEL_URL="postgresql://graph:graph@127.0.0.1:5432/graph-test" + +# Claude: Run runner tests +cargo test -p graph-tests --test runner_tests -- --nocapture +``` + +**Integration Tests:** +```bash +# Human: Start all services for integration tests in a separate terminal +# PostgreSQL: localhost:3011, IPFS: localhost:3001, Anvil: localhost:3021 +nix run .#integration + +# Claude: Run integration tests +cargo test -p graph-tests --test integration_tests +``` + +**Services Configuration:** +The services are configured to use the test suite default ports for unit- and integration tests respectively. + +| Service | Unit Tests Port | Integration Tests Port | Database/Config | +|---------|-----------------|------------------------|-----------------| +| PostgreSQL | 5432 | 3011 | `graph-test` / `graph-node` | +| IPFS | 5001 | 3001 | Data in `./.data/unit` or `./.data/integration` | +| Anvil (Ethereum) | - | 3021 | Deterministic test chain | + +**Service Configuration:** +The setup combines built-in services-flake services with custom multiService modules: + +**Built-in Services:** +- **PostgreSQL**: Uses services-flake's postgres service with a helper function (`mkPostgresConfig`) that provides graph-specific defaults including required extensions. + +**Custom Services** (located in `./nix`): +- `ipfs.nix`: IPFS (kubo) with automatic initialization and configurable ports +- `anvil.nix`: Ethereum test chain with deterministic configuration diff --git a/flake.lock b/flake.lock new file mode 100644 index 00000000000..7f92aea9e23 --- /dev/null +++ b/flake.lock @@ -0,0 +1,178 @@ +{ + "nodes": { + "flake-parts": { + "inputs": { + "nixpkgs-lib": "nixpkgs-lib" + }, + "locked": { + "lastModified": 1754487366, + "narHash": "sha256-pHYj8gUBapuUzKV/kN/tR3Zvqc7o6gdFB9XKXIp1SQ8=", + "owner": "hercules-ci", + "repo": "flake-parts", + "rev": "af66ad14b28a127c5c0f3bbb298218fc63528a18", + "type": "github" + }, + "original": { + "owner": "hercules-ci", + "repo": "flake-parts", + "type": "github" + } + }, + "flake-utils": { + "locked": { + "lastModified": 1644229661, + "narHash": "sha256-1YdnJAsNy69bpcjuoKdOYQX0YxZBiCYZo4Twxerqv7k=", + "owner": "numtide", + "repo": "flake-utils", + "rev": "3cecb5b042f7f209c56ffd8371b2711a290ec797", + "type": "github" + }, + "original": { + "owner": "numtide", + "repo": "flake-utils", + "type": "github" + } + }, + "foundry": { + "inputs": { + "flake-utils": "flake-utils", + "nixpkgs": "nixpkgs" + }, + "locked": { + "lastModified": 1754989872, + "narHash": "sha256-RTl+0GmWIoQium3VN+HjHl2wbZjnRLz5kawo6PMpOYc=", + "owner": "shazow", + "repo": "foundry.nix", + "rev": "6eebe42cec5740c1fbcf2c32ce49f2f8d0e992a9", + "type": "github" + }, + "original": { + "owner": "shazow", + "ref": "stable", + "repo": "foundry.nix", + "type": "github" + } + }, + "nixpkgs": { + "locked": { + "lastModified": 1666753130, + "narHash": "sha256-Wff1dGPFSneXJLI2c0kkdWTgxnQ416KE6X4KnFkgPYQ=", + "owner": "NixOS", + "repo": "nixpkgs", + "rev": "f540aeda6f677354f1e7144ab04352f61aaa0118", + "type": "github" + }, + "original": { + "id": "nixpkgs", + "type": "indirect" + } + }, + "nixpkgs-lib": { + "locked": { + "lastModified": 1753579242, + "narHash": "sha256-zvaMGVn14/Zz8hnp4VWT9xVnhc8vuL3TStRqwk22biA=", + "owner": "nix-community", + "repo": "nixpkgs.lib", + "rev": "0f36c44e01a6129be94e3ade315a5883f0228a6e", + "type": "github" + }, + "original": { + "owner": "nix-community", + "repo": "nixpkgs.lib", + "type": "github" + } + }, + "nixpkgs_2": { + "locked": { + "lastModified": 1755175540, + "narHash": "sha256-V0j2S1r25QnbqBLzN2Rg/dKKil789bI3P3id7bDPVc4=", + "owner": "nixos", + "repo": "nixpkgs", + "rev": "a595dde4d0d31606e19dcec73db02279db59d201", + "type": "github" + }, + "original": { + "owner": "nixos", + "ref": "nixpkgs-unstable", + "repo": "nixpkgs", + "type": "github" + } + }, + "nixpkgs_3": { + "locked": { + "lastModified": 1744536153, + "narHash": "sha256-awS2zRgF4uTwrOKwwiJcByDzDOdo3Q1rPZbiHQg/N38=", + "owner": "NixOS", + "repo": "nixpkgs", + "rev": "18dd725c29603f582cf1900e0d25f9f1063dbf11", + "type": "github" + }, + "original": { + "owner": "NixOS", + "ref": "nixpkgs-unstable", + "repo": "nixpkgs", + "type": "github" + } + }, + "process-compose-flake": { + "locked": { + "lastModified": 1749418557, + "narHash": "sha256-wJHHckWz4Gvj8HXtM5WVJzSKXAEPvskQANVoRiu2w1w=", + "owner": "Platonic-Systems", + "repo": "process-compose-flake", + "rev": "91dcc48a6298e47e2441ec76df711f4e38eab94e", + "type": "github" + }, + "original": { + "owner": "Platonic-Systems", + "repo": "process-compose-flake", + "type": "github" + } + }, + "root": { + "inputs": { + "flake-parts": "flake-parts", + "foundry": "foundry", + "nixpkgs": "nixpkgs_2", + "process-compose-flake": "process-compose-flake", + "rust": "rust", + "services-flake": "services-flake" + } + }, + "rust": { + "inputs": { + "nixpkgs": "nixpkgs_3" + }, + "locked": { + "lastModified": 1755225702, + "narHash": "sha256-i7Rgs943NqX0RgQW0/l1coi8eWBj3XhxVggMpjjzTsk=", + "owner": "oxalica", + "repo": "rust-overlay", + "rev": "4abaeba6b176979be0da0195b9e4ce86bc501ae4", + "type": "github" + }, + "original": { + "owner": "oxalica", + "repo": "rust-overlay", + "type": "github" + } + }, + "services-flake": { + "locked": { + "lastModified": 1755091348, + "narHash": "sha256-aJI3n1CO3ib6CtK1/xuUyfIZM0Lp2zdCFMsFYjjfgQo=", + "owner": "juspay", + "repo": "services-flake", + "rev": "0a3805ceced9f02254d472cc71839811fae85357", + "type": "github" + }, + "original": { + "owner": "juspay", + "repo": "services-flake", + "type": "github" + } + } + }, + "root": "root", + "version": 7 +} diff --git a/flake.nix b/flake.nix new file mode 100644 index 00000000000..3641320cf38 --- /dev/null +++ b/flake.nix @@ -0,0 +1,157 @@ +{ + inputs = { + nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-unstable"; + rust.url = "github:oxalica/rust-overlay"; + foundry.url = "github:shazow/foundry.nix/stable"; + process-compose-flake.url = "github:Platonic-Systems/process-compose-flake"; + services-flake.url = "github:juspay/services-flake"; + flake-parts.url = "github:hercules-ci/flake-parts"; + }; + + outputs = inputs@{ flake-parts, process-compose-flake, services-flake, nixpkgs, rust, foundry, ... }: + flake-parts.lib.mkFlake { inherit inputs; } { + imports = [ process-compose-flake.flakeModule ]; + systems = [ + "x86_64-linux" # 64-bit Intel/AMD Linux + "aarch64-linux" # 64-bit ARM Linux + "x86_64-darwin" # 64-bit Intel macOS + "aarch64-darwin" # 64-bit ARM macOS + ]; + + perSystem = { config, self', inputs', pkgs, system, ... }: + let + overlays = [ + (import rust) + (self: super: { + rust-toolchain = super.rust-bin.stable.latest.default; + }) + foundry.overlay + ]; + + pkgsWithOverlays = import nixpkgs { + inherit overlays system; + }; + in { + devShells.default = pkgsWithOverlays.mkShell { + packages = (with pkgsWithOverlays; [ + rust-toolchain + foundry-bin + solc + protobuf + uv + cmake + corepack + nodejs + postgresql + ]); + }; + + process-compose = let + inherit (services-flake.lib) multiService; + ipfs = multiService ./nix/ipfs.nix; + anvil = multiService ./nix/anvil.nix; + + # Helper function to create postgres configuration with graph-specific defaults + mkPostgresConfig = { name, port, user, password, database, dataDir }: { + enable = true; + inherit port dataDir; + initialScript = { + before = '' + CREATE USER \"${user}\" WITH PASSWORD '${password}' SUPERUSER; + ''; + }; + initialDatabases = [ + { + inherit name; + schemas = [ (pkgsWithOverlays.writeText "init-${name}.sql" '' + CREATE EXTENSION IF NOT EXISTS pg_trgm; + CREATE EXTENSION IF NOT EXISTS btree_gist; + CREATE EXTENSION IF NOT EXISTS postgres_fdw; + CREATE EXTENSION IF NOT EXISTS pg_stat_statements; + GRANT USAGE ON FOREIGN DATA WRAPPER postgres_fdw TO "${user}"; + ALTER DATABASE "${database}" OWNER TO "${user}"; + '') ]; + } + ]; + settings = { + shared_preload_libraries = "pg_stat_statements"; + log_statement = "all"; + default_text_search_config = "pg_catalog.english"; + max_connections = 200; + }; + }; + in { + # Unit tests configuration + unit = { + imports = [ + services-flake.processComposeModules.default + ipfs + anvil + ]; + + cli = { + environment.PC_DISABLE_TUI = true; + options = { + port = 8881; + }; + }; + + services.postgres."postgres-unit" = mkPostgresConfig { + name = "graph-test"; + port = 5432; + dataDir = "./.data/unit/postgres"; + user = "graph"; + password = "graph"; + database = "graph-test"; + }; + + services.ipfs."ipfs-unit" = { + enable = true; + dataDir = "./.data/unit/ipfs"; + port = 5001; + gateway = 8080; + }; + }; + + # Integration tests configuration + integration = { + imports = [ + services-flake.processComposeModules.default + ipfs + anvil + ]; + + cli = { + environment.PC_DISABLE_TUI = true; + options = { + port = 8882; + }; + }; + + services.postgres."postgres-integration" = mkPostgresConfig { + name = "graph-node"; + port = 3011; + dataDir = "./.data/integration/postgres"; + user = "graph-node"; + password = "let-me-in"; + database = "graph-node"; + }; + + services.ipfs."ipfs-integration" = { + enable = true; + dataDir = "./.data/integration/ipfs"; + port = 3001; + gateway = 3002; + }; + + services.anvil."anvil-integration" = { + enable = true; + package = pkgsWithOverlays.foundry-bin; + port = 3021; + timestamp = 1743944919; + }; + }; + }; + }; + }; +} diff --git a/nix/anvil.nix b/nix/anvil.nix new file mode 100644 index 00000000000..df5623aac70 --- /dev/null +++ b/nix/anvil.nix @@ -0,0 +1,42 @@ +{ pkgs, lib, name, config, ... }: +{ + options = { + package = lib.mkOption { + type = lib.types.package; + description = "Foundry package containing anvil"; + }; + + port = lib.mkOption { + type = lib.types.port; + default = 8545; + description = "Port for Anvil RPC server"; + }; + + timestamp = lib.mkOption { + type = lib.types.int; + default = 1743944919; + description = "Timestamp for the genesis block"; + }; + }; + + config = { + outputs.settings.processes.${name} = { + command = "${lib.getExe' config.package "anvil"} --disable-block-gas-limit --disable-code-size-limit --base-fee 1 --block-time 2 --timestamp ${toString config.timestamp} --port ${toString config.port}"; + + availability = { + restart = "always"; + }; + + readiness_probe = { + exec = { + command = "nc -z localhost ${toString config.port}"; + }; + initial_delay_seconds = 3; + period_seconds = 2; + timeout_seconds = 5; + success_threshold = 1; + failure_threshold = 10; + }; + }; + }; +} diff --git a/nix/ipfs.nix b/nix/ipfs.nix new file mode 100644 index 00000000000..72e60725ced --- /dev/null +++ b/nix/ipfs.nix @@ -0,0 +1,54 @@ +{ pkgs, lib, name, config, ... }: +{ + options = { + package = lib.mkPackageOption pkgs "kubo" { }; + + port = lib.mkOption { + type = lib.types.port; + default = 5001; + description = "Port for IPFS API"; + }; + + gateway = lib.mkOption { + type = lib.types.port; + default = 8080; + description = "Port for IPFS gateway"; + }; + }; + + config = { + outputs.settings.processes.${name} = { + command = '' + export IPFS_PATH="${config.dataDir}" + if [ ! -f "${config.dataDir}/config" ]; then + mkdir -p "${config.dataDir}" + ${lib.getExe config.package} init + ${lib.getExe config.package} config Addresses.API /ip4/127.0.0.1/tcp/${toString config.port} + ${lib.getExe config.package} config Addresses.Gateway /ip4/127.0.0.1/tcp/${toString config.gateway} + fi + ${lib.getExe config.package} daemon --offline + ''; + + environment = { + IPFS_PATH = config.dataDir; + }; + + availability = { + restart = "always"; + }; + + readiness_probe = { + http_get = { + host = "localhost"; + port = config.port; + path = "/version"; + }; + initial_delay_seconds = 5; + period_seconds = 3; + timeout_seconds = 10; + success_threshold = 1; + failure_threshold = 10; + }; + }; + }; +}

AltStyle γ«γ‚ˆγ£γ¦ε€‰ζ›γ•γ‚ŒγŸγƒšγƒΌγ‚Έ (->γ‚ͺγƒͺγ‚ΈγƒŠγƒ«) /