Spooled Cloud

Open-source webhook queue & background job infrastructure

License Go SDK Node SDK Python SDK PHP SDK

What is Spooled?

Spooled is like a super-reliable to-do list for your servers.

Imagine you have a website that needs to:

Send welcome emails when users sign up
Process payments from Stripe
Generate PDF reports
Resize uploaded images

These tasks can fail (email server down, payment timeout, etc.). Spooled makes sure they always get done — with automatic retries, scheduling, and real-time monitoring.

┌──────────────────────────────────────────────────────────────────────────┐
│ │
│ YOUR APP SPOOLED YOUR WORKERS │
│ ──────── ─────── ──────────── │
│ │
│ ┌──────────┐ ┌──────────────────┐ ┌─────────────┐ │
│ │ User │───────▶│ Job Queue │──────────▶│ Worker │ │
│ │ signs │ Enqueue│ │ Claim │ sends │ │
│ │ up │ job │ ┌───┐┌───┐┌───┐ │ job │ welcome │ │
│ └──────────┘ │ │ J ││ J ││ J │ │ │ email │ │
│ │ └───┘└───┘└───┘ │ └──────┬──────┘ │
│ │ │ │ │
│ ┌──────────┐ │ * Retries │ ┌──────▼──────┐ │
│ │ Stripe │───────▶│ * Priority │ │ DONE! │ │
│ │ webhook │ │ * Schedules │ │ or │ │
│ └──────────┘ │ * Dead-letter │ │ Retry │ │
│ └──────────────────┘ └─────────────┘ │
│ │
└──────────────────────────────────────────────────────────────────────────┘

Key Features

Feature	Description
Job Queues	Reliable FIFO queues with priority support (0-100)
Automatic Retries	Configurable retry policies with exponential backoff
Schedules	Cron-based scheduling with timezone support (6-field with seconds!)
Workflows	DAG-based job dependencies (run job B after job A completes)
Webhooks	Receive webhooks from Stripe, GitHub, etc. → auto-queue as jobs
Real-time Dashboard	See all your jobs, queues, and workers live
High Performance	gRPC streaming for workers processing 1000s of jobs/sec
Dead Letter Queue	Failed jobs are saved for inspection and manual retry

Quick Start (5 minutes)

Step 1: Get an API Key

Go to dashboard.spooled.cloud
Sign up / Log in
Copy your API key (starts with sk_live_... or sk_test_...)

Step 2: Create Your First Job

Choose your language:

🟢 Node.js / TypeScript

npm install @spooled/sdk

import { SpooledClient } from "@spooled/sdk";
const client = new SpooledClient({ 
 apiKey: "sk_live_YOUR_API_KEY" 
});
// Create a job
const job = await client.jobs.create({
 queueName: "emails",
 payload: { 
 to: "user@example.com", 
 subject: "Welcome!" 
 }
});
console.log("Job created:", job.id);

🐍 Python

pip install spooled

from spooled import SpooledClient
client = SpooledClient(api_key="sk_live_YOUR_API_KEY")
# Create a job
job = client.jobs.create({
 "queue_name": "emails",
 "payload": {
 "to": "user@example.com",
 "subject": "Welcome!"
 }
})
print(f"Job created: {job.id}")
client.close()

🔵 Go

go get github.com/spooled-cloud/spooled-sdk-go

package main
import (
 "context"
 "fmt"
 "github.com/spooled-cloud/spooled-sdk-go/spooled"
 "github.com/spooled-cloud/spooled-sdk-go/spooled/resources"
)
func main() {
 client := spooled.NewClient(
 spooled.WithAPIKey("sk_live_YOUR_API_KEY"),
 )
 job, _ := client.Jobs().Create(context.Background(), &resources.CreateJobRequest{
 QueueName: "emails",
 Payload: map[string]any{
 "to": "user@example.com",
 "subject": "Welcome!",
 },
 })
 fmt.Println("Job created:", job.ID)
}

🐘 PHP

composer require spooled-cloud/spooled

<?php
use Spooled\SpooledClient;
use Spooled\Config\ClientOptions;
$client = new SpooledClient(new ClientOptions(
 apiKey: 'sk_live_YOUR_API_KEY'
));
$job = $client->jobs->create([
 'queue' => 'emails',
 'payload' => [
 'to' => 'user@example.com',
 'subject' => 'Welcome!'
 ]
]);
echo "Job created: {$job->id}\n";

🌐 cURL (REST API)

curl -X POST https://api.spooled.cloud/api/v1/jobs \
 -H "Authorization: Bearer sk_live_YOUR_API_KEY" \
 -H "Content-Type: application/json" \
 -d '{
 "queue_name": "emails",
 "payload": {
 "to": "user@example.com",
 "subject": "Welcome!"
 }
 }'

Step 3: Process Jobs (Worker)

Your worker claims jobs, processes them, and marks them complete:

🟢 Node.js Worker

import { SpooledWorker } from "@spooled/sdk";
const worker = new SpooledWorker({
 apiKey: "sk_live_YOUR_API_KEY",
 queueName: "emails",
 handler: async (job) => {
 // Your business logic here
 console.log("Processing:", job.payload);
 
 await sendEmail(job.payload.to, job.payload.subject);
 
 return { success: true };
 }
});
worker.start();
console.log("🚀 Worker started, waiting for jobs...");

🐍 Python Worker

from spooled import SpooledWorker
def handle_job(job):
 print(f"Processing: {job.payload}")
 send_email(job.payload["to"], job.payload["subject"])
 return {"success": True}
worker = SpooledWorker(
 api_key="sk_live_YOUR_API_KEY",
 queue_name="emails",
 handler=handle_job
)
print("🚀 Worker started, waiting for jobs...")
worker.start()

🔵 Go Worker

package main
import (
 "context"
 "fmt"
 "github.com/spooled-cloud/spooled-sdk-go/spooled"
 "github.com/spooled-cloud/spooled-sdk-go/spooled/worker"
)
func main() {
 w := spooled.NewSpooledWorker(
 "sk_live_YOUR_API_KEY",
 "emails",
 func(ctx context.Context, job *worker.Job) (map[string]any, error) {
 fmt.Println("Processing:", job.Payload)
 // Your business logic here
 return map[string]any{"success": true}, nil
 },
 )
 fmt.Println("🚀 Worker started, waiting for jobs...")
 w.Start(context.Background())
}

Architecture Overview

 ┌──────────────────────────────────────────┐
 │ SPOOLED CLOUD │
 │ │
 ┌──────────┐ │ ┌────────────────────────────────┐ │ ┌──────────┐
 │ Your │ REST │ │ API Server │ │ REST │ Your │
 │ App │───────▶│ │ (Rust, high-performance) │◀───┼────────│ Workers │
 └──────────┘ │ │ │ │ └──────────┘
 │ │ * Job management │ │
 ┌──────────┐ │ │ * Queue operations │ │ ┌──────────┐
 │ Stripe │ HTTP │ │ * Webhook ingestion │ │ gRPC │ High-vol │
 │ GitHub │───────▶│ │ * Schedule execution │ │◀───────│ Workers │
 │ etc. │ │ └───────────────┬────────────────┘ │ └──────────┘
 └──────────┘ │ │ │
 │ ▼ │
 │ ┌────────────────────────────────┐ │
 │ │ PostgreSQL + Redis │ │
 │ │ (Durable storage + caching) │ │
 │ └────────────────────────────────┘ │
 │ │
 │ ┌────────────────────────────────┐ │ ┌──────────┐
 │ │ Dashboard │ │ WS/ │ You │
 │ │ (Real-time job monitoring) │◀───┼──SSE───│ watching │
 │ └────────────────────────────────┘ │ └──────────┘
 └──────────────────────────────────────────┘

What's in This Repository

This is a monorepo containing the entire Spooled platform:

spooled-cloud/
├── spooled-backend/ # Rust API server (REST + gRPC + WebSocket)
├── spooled-dashboard/ # React dashboard (job monitoring, billing)
├── spooled-frontend/ # Astro marketing site + documentation
├── spooled-sdk-nodejs/ # Node.js/TypeScript SDK
├── spooled-sdk-python/ # Python SDK
├── spooled-sdk-go/ # Go SDK
└── spooled-sdk-php/ # PHP SDK

SDK Status

SDK	Package	Status	Tests
Node.js	`@spooled/sdk`	✅ Production Ready	175 tests
Python	`spooled`	✅ Production Ready	173 tests
Go	`github.com/spooled-cloud/spooled-sdk-go`	✅ Production Ready	194 tests
PHP	`spooled-cloud/spooled`	✅ Production Ready	175 tests

Authentication

API Key Formats

Prefix	Environment	Usage
`sk_live_...`	Production	Real data, billing enabled
`sk_test_...`	Test/Dev	Safe for testing, no billing

Note: Documentation uses sp_live_/sp_test_ prefixes to avoid GitHub secret scanning. All SDKs accept both formats.

How to Authenticate

REST API — Use Bearer token in header:

Authorization: Bearer sk_live_YOUR_API_KEY

gRPC API — Use metadata:

x-api-key: sk_live_YOUR_API_KEY

Production Endpoints

Service	URL	Protocol
Website & Docs	https://spooled.cloud	HTTPS
Dashboard	https://dashboard.spooled.cloud	HTTPS
REST API	https://api.spooled.cloud	HTTPS
gRPC API	grpc.spooled.cloud:443	gRPC + TLS

Local Development

Prerequisites

Tool	Version	Purpose
Docker	Latest	Database & cache containers
Rust	1.70+	Backend server
Node.js	20+	Dashboard & frontend
Go	1.21+	Go SDK
Python	3.11+	Python SDK
PHP	8.2+	PHP SDK

Start the Backend

# 1. Clone and enter the repo
git clone https://github.com/Spooled-Cloud/spooled-cloud.git
cd spooled-cloud/spooled-backend
# 2. Start infrastructure (Postgres, Redis)
docker compose up -d postgres redis pgbouncer
# 3. Run the API server
cargo run
# REST API available at http://localhost:8080
# gRPC API available at localhost:50051

Start the Dashboard

cd spooled-dashboard
PUBLIC_API_URL=http://localhost:8080 \
PUBLIC_WS_URL=ws://localhost:8080 \
docker compose up -d
# Dashboard available at http://localhost:4321

Start the Docs Site

cd spooled-frontend
npm install
npm run dev -- --port 4322
# Docs available at http://localhost:4322

Local Ports Reference

Service	Port	Environment Variable
Backend REST/WS/SSE	8080	`PORT`
Backend gRPC	50051	`GRPC_PORT`
Dashboard	4321	`PORT`
Docs (Astro)	4322	Use `--port` flag
PostgreSQL	5433	Docker mapped
PgBouncer	6432	Docker mapped
Redis	6379	Docker mapped

Running Tests

# Backend tests (Rust)
cd spooled-backend && cargo test
# Frontend tests
cd spooled-frontend && npm test && npm run lint
# SDK integration tests (the "golden" reference tests)
cd spooled-sdk-nodejs && npm test
cd spooled-sdk-python && python -m pytest
cd spooled-sdk-go && go test ./...
cd spooled-sdk-php && ./vendor/bin/phpunit

Common Use Cases

1. Process Stripe Webhooks

// Stripe sends webhook → Spooled queues it → Your worker processes it
const job = await client.jobs.create({
 queueName: "stripe-webhooks",
 payload: stripeEvent,
 idempotencyKey: stripeEvent.id, // Prevents duplicates!
});

2. Send Emails in Background

// Don't make users wait for email sending
const job = await client.jobs.create({
 queueName: "emails",
 payload: { to: user.email, template: "welcome" },
 priority: 10, // Higher = processed first
});

3. Schedule Daily Reports

// Run every day at 9 AM EST
const schedule = await client.schedules.create({
 name: "daily-report",
 queueName: "reports",
 cronExpression: "0 0 9 * * *", // 6-field cron (with seconds!)
 timezone: "America/New_York",
 payload: { type: "daily-summary" },
});

4. Workflow Dependencies

// Job B runs only after Job A completes
const workflow = await client.workflows.create({
 name: "user-onboarding",
 jobs: [
 { id: "create-account", queueName: "accounts", payload: {...} },
 { id: "send-welcome", queueName: "emails", payload: {...}, dependsOn: ["create-account"] },
 { id: "setup-billing", queueName: "billing", payload: {...}, dependsOn: ["create-account"] },
 ]
});

Contributing

We love contributions! See the contribution guides:

Key Files to Update

When changing behavior, remember to update:

SDK test-local.* scripts (canonical usage examples)
spooled-frontend/src/lib/snippets.ts (docs code snippets)

Documentation

Resource	Location
Product Docs	`spooled-frontend/src/pages/docs/*`
OpenAPI Spec	`spooled-backend/docs/openapi.yaml`
Backend Guides	`spooled-backend/docs/guides/*`
Stripe Setup	`STRIPE_SETUP.md`

License

Apache 2.0 — See LICENSE for details.

Support the Project

If Spooled saves you time or prevents webhook outages:

⭐ Star this repo — It helps others find us!
💝 Sponsor on GitHub — Support development

Built with ❤️ for developers who hate losing webhooks

Website • Dashboard • Documentation

Spooled Cloud

Open-source webhook queue & background job infrastructure

What is Spooled?

Key Features

Quick Start (5 minutes)

Step 1: Get an API Key

Step 2: Create Your First Job

Step 3: Process Jobs (Worker)

Architecture Overview

What's in This Repository

SDK Status

Authentication

API Key Formats

How to Authenticate

Production Endpoints

Local Development

Prerequisites

Start the Backend

Start the Dashboard

Start the Docs Site

Local Ports Reference

Running Tests

Common Use Cases

1. Process Stripe Webhooks

2. Send Emails in Background

3. Schedule Daily Reports

4. Workflow Dependencies

Contributing

Key Files to Update

Documentation

License

Support the Project

Popular repositories Loading

Repositories

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

People

Top languages

Uh oh!

Most used topics

Uh oh!