An async-first, FastAPI-based microservice that queues and processes media generation jobs using a mocked Replicate API. It utilizes Celery for background task management, PostgreSQL for job tracking, RabbitMQ as the message broker, and S3 storage for file uploads.
- FastAPI – API layer
- SQLAlchemy & Alembic - Async ORM and Migrations
- Celery – Async task processing
- RabbitMQ – Message broker
- PostgreSQL – Database
- S3 (via boto3) – Cloud Object Storage
- Docker Compose – Containerization/Deployments
flowchart LR
A((Mediator API))
B[(PostgreSQL)]
C(RabbitMQ)
D(Celery)
E[(S3 Bucket)]
F(Client)
G((Mocked Replicate API))
F -->|Generation Requests| A
A -->|Queue Tasks| C --> D
A -->|Persist Job Metadata| B
D --->|Submit Jobs| G
G -->|Webhooks| A
D -->|Upload Media Files| E
git clone https://github.com/Robin-07/mediator
cd mediatorDATABASE_URL=postgresql+asyncpg://postgres:postgres@db:5432/postgres CELERY_BROKER_URL=amqp://guest:guest@rabbitmq:5672// # AWS settings AWS_ACCESS_KEY_ID=AKIAVR**** AWS_SECRET_ACCESS_KEY=5/WNM24++BDWy**** AWS_REGION=us-east-1 S3_BUCKET_NAME=media-generations # Application settings DUMMY_IMAGE_URL=https://upload.wikimedia.org/wikipedia/commons/thumb/4/4d/Cat_November_2010-1a.jpg/1200px-Cat_November_2010-1a.jpg REPLICATE_PREDICTIONS_MOCK_URL=http://mediator:8000/api/v1/predictions REPLICATE_MODEL_VERSION=test-model REPLICATE_CALLBACK_URL=http://mediator:8000/api/v1/callback DEBUG=1
docker compose up --build
# Enter running mediator container docker compose exec mediator bash # Run following commands alembic revision --autogenerate -m "your message" alembic upgrade head
- Use actual Replicate API
- Rate limiting & robust retry logic
- Webhook validation
- Unit & Integration tests
- Prometheus/Grafana integration for metrics
- Add documentation