Self-hosted, open-source platform for running Apify Actors on your own infrastructure.
License: MIT TypeScript Node.js
Crawlee Cloud - Self-hosted platform for running web scrapers | Product Hunt
Love the Crawlee/Apify ecosystem but want the freedom to run things your way? Crawlee Cloud brings the same great developer experience to your own infrastructure. Keep using the tools you love — just host them wherever you want.
- � Your infrastructure — Deploy on your own servers, cloud, or anywhere you like
- 🔒 Complete privacy — Your data stays exactly where you want it
- ⚡ SDK compatible — Works seamlessly with the Apify SDK you already know
- 🐳 Container-based — Each Actor runs in an isolated Docker container
- 📊 Beautiful dashboard — Monitor runs, explore datasets, manage everything visually
# Instead of pointing to Apify's servers... export APIFY_API_BASE_URL=https://api.apify.com/v2 # Point to your own Crawlee Cloud instance export APIFY_API_BASE_URL=https://your-server.com/v2 export APIFY_TOKEN=your-token
Your existing Actor code works without any modifications:
import { Actor } from 'apify'; await Actor.init(); await Actor.pushData({ title: 'Scraped data' }); await Actor.exit();
- Node.js 18+
- Docker & Docker Compose
- PostgreSQL, Redis, and S3-compatible storage (or use our Docker setup)
git clone https://github.com/crawlee-cloud/crawlee-cloud.git
cd crawlee-cloud
npm install# Starts PostgreSQL, Redis, and MinIO
npm run docker:devcp .env.example .env
# Edit .env with your settingsnpm run build npm run db:migrate npm run dev
The API server starts at http://localhost:3000.
┌─────────────────────────────────────────────────────────────────┐
│ Your Actors │
│ (using official Apify SDK, no changes) │
└────────────────────────────┬────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Crawlee Cloud API │
│ (Apify-compatible REST endpoints) │
└─────────────────────────────────────────────────────────────────┘
│ │ │
▼ ▼ ▼
┌──────────┐ ┌─────────┐ ┌─────────┐
│PostgreSQL│ │ Redis │ │ S3/MinIO│
│ metadata │ │ queues │ │ blobs │
└──────────┘ └─────────┘ └─────────┘
| Component | Description |
|---|---|
| API Server | Fastify-based REST API compatible with Apify's v2 endpoints |
| Runner | Polls job queue and executes Actors in Docker containers |
| Dashboard | Next.js web UI for monitoring and management |
| CLI | Command-line tool for pushing and running Actors |
| Guide | Description |
|---|---|
| API Reference | REST API endpoints and usage |
| CLI Guide | Command-line interface |
| Dashboard | Web interface overview |
| Deployment | Production deployment guide |
| Runner | Actor execution engine |
| SDK Compatibility | Apify SDK integration |
| Feature | Status |
|---|---|
Datasets (Actor.pushData) |
✅ Supported |
Key-Value Stores (Actor.getValue/setValue) |
✅ Supported |
| Request Queues | ✅ Supported |
| Request deduplication | ✅ Supported |
| Distributed locking | ✅ Supported |
We welcome contributions! Please see our Contributing Guide for details.
# Run tests npm test # Type checking npm run typecheck # Linting npm run lint
This project is licensed under the MIT License.
Built with ❤️ for the web scraping community