Streaming Service

Go Version License

A high-performance, production-ready video/audio streaming service built in Go, designed for 100M+ RPS with HLS adaptive bitrate streaming.

🎯 Overview

This streaming platform provides enterprise-grade video delivery with:

HLS Adaptive Streaming - Automatic quality adjustment based on viewer bandwidth
Multi-Bitrate Encoding - 1080p, 720p, 480p, 360p renditions
Audio Support - Standalone audio streaming and video audio extraction
AWS Native - S3 storage, DynamoDB metadata, CloudFront CDN
Kubernetes Ready - Terraform-managed infrastructure for auto-scaling

🏗️ Architecture

┌─────────────────────────────────────────────────────────────────────────────┐
│ CLIENT LAYER │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Web │ │ Mobile │ │ Smart TV │ │ OTT │ │
│ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ │
└───────┼─────────────┼─────────────┼─────────────┼───────────────────────────┘
 │ │ │ │
 └─────────────┴──────┬──────┴─────────────┘
 ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ CDN / EDGE LAYER │
│ ┌─────────────────┐ │
│ │ CloudFront │ ◄── Cached HLS Segments │
│ └────────┬────────┘ │
└─────────────────────────────────┼───────────────────────────────────────────┘
 ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ API GATEWAY LAYER │
│ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │
│ │ Load Balancer │──│ API Server │──│ Auth │ │
│ └───────────────┘ └───────┬───────┘ └───────────────┘ │
│ │ │
└─────────────────────────────┼───────────────────────────────────────────────┘
 ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ MICROSERVICES LAYER │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Upload │ │ Transcode │ │ Stream │ │ Audio │ │
│ │ Service │ │ Service │ │ Service │ │ Service │ │
│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │
└─────────┼────────────────┼────────────────┼────────────────┼────────────────┘
 │ │ │ │
 ▼ ▼ ▼ ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ DATA & PROCESSING LAYER │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ S3 │ │ FFMPEG │ │ DynamoDB │ │ Redis │ │
│ │ (Storage) │ │ (Encoding) │ │ (Metadata) │ │ (Queue) │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘

📁 Project Structure

streaming-service/
├── cmd/
│ ├── api/ # API server entrypoint
│ └── worker/ # Transcoding worker entrypoint
├── internal/
│ ├── api/ # HTTP handlers & Chi router
│ ├── config/ # Viper configuration management
│ ├── domain/ # Business entities (Media, Video, Audio)
│ ├── media/
│ │ ├── ffmpeg/ # FFMPEG video/audio processors
│ │ └── processor/ # Factory & Strategy pattern implementations
│ ├── queue/ # Redis job queue with priority support
│ ├── repository/
│ │ ├── dynamodb/ # Metadata CRUD operations
│ │ └── s3/ # Object storage with presigned URLs
│ └── service/
│ ├── audio/ # Audio extraction & processing
│ ├── stream/ # Playback URL generation
│ ├── transcode/ # HLS transcoding pipeline
│ └── upload/ # File upload handling
├── pkg/
│ └── logger/ # Zap structured logging
├── deployments/
│ ├── docker/ # Multi-stage Dockerfiles
│ ├── kubernetes/ # K8s manifests
│ └── terraform/ # Infrastructure as Code
└── config.yaml # Application configuration

🎨 Design Patterns

Factory Pattern

Creates appropriate media processors based on content type:

processor := factory.CreateProcessor(domain.MediaTypeVideo) // or MediaTypeAudio

Strategy Pattern

Interchangeable transcoding strategies for different output formats:

executor.AddStrategy(NewHLSTranscodeStrategy(profile1080p))
executor.AddStrategy(NewHLSTranscodeStrategy(profile720p))
executor.Execute(ctx, input, outputDir, cmdExecutor)

Worker Pool Pattern

Concurrent job processing with configurable parallelism:

worker := transcode.NewWorker(queue, service, concurrency, logger)
worker.Start(ctx)

🚀 Quick Start

Prerequisites

Go 1.22+
Docker & Docker Compose
FFMPEG (for local development)
AWS CLI (configured)

Local Development

# Clone repository
git clone https://github.com/ichbingautam/streaming-service.git
cd streaming-service
# Install dependencies
go mod download
# Start infrastructure (Redis + LocalStack)
docker-compose up -d redis localstack
# Run API server
make run-api
# Run transcoding worker (in another terminal)
make run-worker

Docker Compose (Full Stack)

# Start all services
docker-compose up -d
# View logs
docker-compose logs -f api worker

📡 API Endpoints

Method	Endpoint	Description
`GET`	`/health`	Health check
`GET`	`/ready`	Readiness probe
`POST`	`/api/v1/upload`	Upload media file (multipart)
`POST`	`/api/v1/upload/presign`	Get presigned upload URL
`POST`	`/api/v1/upload/{id}/confirm`	Confirm presigned upload
`GET`	`/api/v1/media`	List user's media
`GET`	`/api/v1/media/{id}`	Get media details
`DELETE`	`/api/v1/media/{id}`	Delete media
`GET`	`/api/v1/media/{id}/playback`	Get HLS playback URL

Example: Upload Video

curl -X POST http://localhost:8080/api/v1/upload \
 -H "X-User-ID: user123" \
 -F "file=@video.mp4" \
 -F "title=My Video" \
 -F "description=Sample video"

Example: Get Playback URL

curl http://localhost:8080/api/v1/media/{media_id}/playback \
 -H "X-User-ID: user123"

⚙️ Configuration

Configuration via config.yaml or environment variables (prefix: STREAM_):

app:
 name: streaming-service
 version: 1.0.0
 environment: production
server:
 port: 8080
 readtimeout: 30s
 writetimeout: 30s
aws:
 region: us-east-1
 s3rawbucket: streaming-raw-media
 s3processedbucket: streaming-processed-media
 dynamodbtable: video-metadata
 cloudfrontdomain: d1234.cloudfront.net
redis:
 host: redis
 port: 6379
ffmpeg:
 binarypath: ffmpeg
 segmentduration: 6
 profiles:
 - name: "1080p"
 width: 1920
 height: 1080
 videobitrate: "5000k"
 audiobitrate: "192k"
worker:
 concurrency: 4
 jobtimeout: 30m
log:
 level: info
 format: json

☸️ Kubernetes Deployment

Using Terraform

cd deployments/terraform
# Initialize Terraform
terraform init
# Review plan
terraform plan -var="environment=production"
# Apply infrastructure
terraform apply -var="environment=production"

Using kubectl

# Create namespace
kubectl create namespace streaming
# Apply manifests
kubectl apply -f deployments/kubernetes/ -n streaming
# Check deployment
kubectl get pods -n streaming

🔧 Development

# Run tests
make test
# Run with coverage
make test-coverage
# Lint code
make lint
# Build binaries
make build
# Build Docker images
make docker-build

📊 Performance Targets

Metric	Target
API Latency (p99)	< 50ms
Transcode Time (1min video)	< 60s
CDN Cache Hit Ratio	> 95%
Concurrent Streams	100M+

🛠️ Technology Stack

Component	Technology
Language	Go 1.22+
HTTP Router	Chi
Configuration	Viper
Logging	Zap
Media Processing	FFMPEG
Storage	AWS S3
Metadata	AWS DynamoDB
CDN	AWS CloudFront
Queue	Redis
Containers	Docker
Orchestration	Kubernetes
IaC	Terraform

📄 License

MIT License - see LICENSE for details.

🤝 Contributing

Fork the repository
Create feature branch (git checkout -b feature/amazing)
Commit changes (git commit -m 'Add amazing feature')
Push to branch (git push origin feature/amazing)
Open Pull Request

Name		Name	Last commit message	Last commit date
.github		.github
cmd		cmd
deployments		deployments
internal		internal
pkg/logger		pkg/logger
.gitignore		.gitignore
.golangci.yml		.golangci.yml
Makefile		Makefile
README.md		README.md
cliff.toml		cliff.toml
config.yaml		config.yaml
docker-compose.yml		docker-compose.yml
go.mod		go.mod
go.sum		go.sum

Navigation Menu

Latest commit

History

Folders and files

Repository files navigation