Moving a stream into a warehouse usually means choosing between two shapes. Take a general-purpose stream processor and you inherit its delivery guarantees and its operational maturity, but your transformations are written in whatever language that runtime accepts, and the runtime is not yours to profile. Write the consumer loop yourself and you get the opposite trade: your language, your allocator, your profile, and every guarantee is now your problem, including the ones you find out about in production.

Spate is the third shape. Transformations are ordinary Rust functions, monomorphized into the pipeline rather than interpreted by it. Delivery, backpressure, checkpointing, rebalancing and drain-on-shutdown belong to the framework, and the properties they hold to are written down, numbered, and tested.

One process runs one pipeline, in four stages. The property each stage holds to is stated and numbered in docs/INVARIANTS.md.

Extract — one consumer per process. Partitions fan out across CPU-pinned threads as zero-copy lanes, so a record is read from the source buffer and never copied on the way in. A thread that cannot keep up pauses its lanes and keeps polling; it never blocks on a channel send, because a blocked poll loop is how a consumer gets evicted from its group.

Transform — operators are stateful closures chained in Rust. A chain compiles to a single loop over borrowed records with no per-record allocation. Record-level failure is Skip or Fail, never a silent drop: both are surfaced through metrics.

Load — sinks are sharded and replicated, running asynchronously on a shared I/O runtime. The chain routes rows into bounded per-shard queues; workers merge chunks, seal batches, rotate replicas and retry. The queue bound is the backpressure signal that reaches all the way back to Extract.

Observe — a source watermark advances only behind data the sink has acknowledged as durable, so commits trail delivery rather than leading it. Instrumentation is built on the metrics facade, so any recorder in that ecosystem works; a Prometheus scrape endpoint and health probes ship on the admin server.

[dependencies]
spate = { version = "0.2", features = ["kafka", "clickhouse", "avro"] }

Nothing is enabled by default. A pipeline that only writes to ClickHouse never compiles the Kafka tree and never resolves rdkafka into its lockfile.

Operators are stateful closures composed into one monomorphized loop; YAML carries the tuning and connector configuration. This is a whole program, against in-memory mocks, so it needs no infrastructure to build or run:

use spate::prelude::*;
use spate_test::{TestDeserializer, TestEncoder, capture_sink, memory_source};
fn main() -> Result<(), Box<dyn std::error::Error>> {
 let config = PipelineConfig::from_str(
 "pipeline: { name: demo, threads: 1 }\n\
 checkpoint: { interval: 100ms }\n\
 source: { memory: {} }\n\
 sink: { capture: {} }",
 )?;
 let (source, _handle) = memory_source();
 let (sink, _script) = capture_sink(1, 1);
 let report = Pipeline::from_config(config)?
 .sink(sink)?
 .chains(|ctx| {
 // The sink's YAML `chunk:` block, bound before `with_metrics`
 // takes ownership of `ctx.pipeline`.
 let chunk_cfg = ctx.chunk();
 chain_owned::<Vec<u8>, _>(TestDeserializer::split_on(b','))
 .with_metrics(ctx.pipeline, "main")
 .filter(|word: &Vec<u8>| !word.is_empty())
 .map(|word: Vec<u8>| word.to_ascii_uppercase())
 .sink(TestEncoder, KeyHashRouter, chunk_cfg, ctx.queues, ctx.budget)
 .build()
 })
 .run(source)?;
 report.log();
 std::process::exit(report.exit_code());
}

Swap memory_source() for KafkaSource::from_component_config and the capture sink for a ClickHouse one, and the chain in the middle does not change. run installs signal handling and blocks until the pipeline has drained; tests use into_runtime instead, which hands back a shutdown handle so they can drive it. A version that scripts records through and asserts on what the sink captured runs from the repository:

cargo run -p spate --example memory_pipeline

Start at the examples index, which lists every example by what it shows, in five tiers, with what each one needs in order to run. kafka_avro_to_clickhouse is the fully-commented production assembly, custom_source_sink is the connector-author tutorial, and s3_coordinated_backfill runs two instances sharing one bounded backfill without either duplicating it. examples/docker covers containers and Kubernetes: probes, drain timeouts, sizing.

At-least-once. A batch's offsets commit only after every record derived from it is durably written (or intentionally dropped by filter/Skip policies). That holds across rebalances, shutdown, and failure, where the watermark stalls rather than ever committing past unacknowledged data. Duplicates remain possible: in-session retries are idempotent where sinks support it (ClickHouse deduplication tokens), but crash replay re-batches with new boundaries and will land rows twice. Design target tables to tolerate that (ReplacingMergeTree with a version column is the sanctioned ClickHouse pattern).

And the framework itself:

Each connector feature turns on one crate. Finer knobs are separate features, listed with what they pull in on docs.rs: a SIMD JSON backend, TLS and SASL for Kafka, chrono/time/uuid/rust_decimal column types for ClickHouse, a NATS JetStream store for coordination. Writing your own connector is a supported path, not a fork: see custom_source_sink.

Single-node throughput is measured. A change that reaches Rust runs allocation assertions and request-shape assertions, and one whose blast radius reaches a benched crate runs instruction-count benches too. Those compare counts rather than elapsed time, so a regression they report is a property of the change and not of how busy the runner was.

Wall-clock benches sit beside them as cargo bench targets, and nothing gates on one: a wall-clock figure is only worth reading against another taken on the same quiet hardware, and a shared CI runner is not that.

proptest covers the checkpoint tracker, the codecs and the assignment protocol across seven crates. loom models the tracker's concurrency directly; that module stays synchronous and free of async runtime types. Kafka runs against librdkafka's MockCluster on every pull request; brokers, ClickHouse and object stores run against real containers whenever a change reaches them, and on a schedule regardless. The work-assignment invariants each name the property test that enforces them.

The most useful contribution is one that proves a delivery guarantee wrong.

The user guide is published at https://spate.kainth.dev/ (source in website/, content in docs/). The API reference is on docs.rs.

• docs/INVARIANTS.md — the numbered properties the engine is arranged around.
• docs/adr/ — one record per architectural decision, with the alternatives that were rejected and why.
• docs/METRICS.md — every metric, its labels, and alerting starting points.
• examples/docker — containers and Kubernetes.

Under active initial development; APIs are not yet stable (0.x). Breaking changes ship in a minor bump and are called out in CHANGELOG.md. The newest 0.x minor is the supported one.

CONTRIBUTING.md has what is worth contributing, how to build and test it, and how changes land. The Code of Conduct applies throughout. AI_POLICY.md covers what a contribution has to withstand, whatever wrote it.

Vulnerabilities go through GitHub's private advisory flow, never a public issue; see SECURITY.md.

Copyright 2026 Marcus Kainth.

Licensed under the Apache License, Version 2.0; see LICENSE.

Dependency licenses are inventoried in THIRD-PARTY.md; the full texts are published at spate.kainth.dev/licenses.

Contributions are accepted under the same terms, per Apache-2.0 §5. There is no CLA to sign.