Autonomous Kubernetes cluster observability with AI-powered weekly health reports
An intelligent Kubernetes monitoring agent that uses Claude AI to autonomously investigate cluster health, analyze metrics from Prometheus, and generate comprehensive weekly PDF reports delivered via Slack.
- π€ AI-Powered Analysis: Claude AI autonomously investigates cluster issues using direct Python tools
- π Prometheus Integration: Analyzes metrics to detect resource inefficiencies (optional)
- π Read-Only by Design: All operations are read-only for safety
- π PDF Reports: Professional HTML reports converted to PDF via WeasyPrint
- π§ Slack Integration: Reports delivered via Slack with detailed tool usage information
- ποΈ Historical Tracking: SQLite storage for report history
- π REST API: FastAPI server for on-demand report generation
- β‘ Graceful Degradation: Works with or without Prometheus
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Kubernetes Cluster β
β β
β ββββββββββββββββββββββββββββββββββββββββββββββββ β
β β K8s Watchdog AI (FastAPI) β β
β β β β
β β ββββββββββββββββββββββββββββββββββββββ β β
β β β Claude AI Agent β β β
β β β - Autonomous investigation β β β
β β β - Tool selection & execution β β β
β β β - Report generation β β β
β β βββββββββββ¬βββββββββββββββββββββββββββ β β
β β β β β
β β βββββββββββΌβββββββββββ βββββββββββββββββββ β
β β β Kubernetes Tools β β Prometheus ββ β
β β β - get pods/nodes β β Tools ββ β
β β β - describe β β - query ββ β
β β β - logs β β - range query ββ β
β β β - events β β - memory/cpu ββ β
β β ββββββββββββββββββββββ βββββββββββββββββββ β
β β β β
β β ββββββββββββββββββββββββββββββββββββββ β β
β β β Report Generator & Storage β β β
β β β - WeasyPrint (HTML β PDF) β β β
β β β - SQLite (history) β β β
β β β - Slack Files API v2 β β β
β β ββββββββββββββββββββββββββββββββββββββ β β
β ββββββββββββββββββββ¬ββββββββββββββββββββββββββββ β
ββββββββββββββββββββββΌββββββββββββββββββββββββββββββββββ
β
βΌ
Slack Webhook
- Kubernetes cluster with kubectl access (or kubeconfig for local development)
- Anthropic API key (Get one here)
- Slack webhook URL (Create one)
- Slack Bot Token and Channel ID for file uploads (Create bot)
- Prometheus running in cluster (optional - reports work without it)
# 1. Clone the repository git clone https://github.com/helmcode/k8s-watchdog-ai.git cd k8s-watchdog-ai # 2. Copy and configure environment cp .env.example .env # Edit .env with your API keys and settings # 3. Run with docker-compose docker-compose up -d # 4. Trigger report generation curl -X POST http://localhost:8000/report # 5. Check status curl http://localhost:8000/health # 6. View logs docker-compose logs -f
# 1. Store secrets in Vault (if using Vault) vault kv put helmcode_platform/k8s_watchdog_ai \ ANTHROPIC_API_KEY="sk-ant-..." \ SLACK_WEBHOOK_URL="https://hooks.slack.com/..." \ SLACK_BOT_TOKEN="xoxb-..." \ SLACK_CHANNEL="C123456789" # 2. Install with Helm helm install k8s-watchdog-ai ./helm \ --namespace watchdog-ai \ --create-namespace \ --values ./helm/values/prod.yaml # 3. Verify deployment kubectl get pods -n watchdog-ai kubectl logs -f deployment/k8s-watchdog-ai -n watchdog-ai
For detailed Helm deployment instructions, see helm/README.md.
# Apply ArgoCD Application kubectl apply -f helm/argocd/application.yaml # Monitor deployment argocd app get k8s-watchdog-ai
For ArgoCD configuration details, see helm/argocd/README.md.
| Variable | Required | Default | Description |
|---|---|---|---|
ANTHROPIC_API_KEY |
β | - | Claude API key |
ANTHROPIC_MODEL |
β | claude-sonnet-4-20250514 | AI model to use |
SLACK_WEBHOOK_URL |
β | - | Slack webhook for messages |
SLACK_BOT_TOKEN |
β | - | Bot token for file uploads |
SLACK_CHANNEL |
β | - | Channel ID (e.g., C123456789) |
PROMETHEUS_URL |
β | http://prometheus:9090 | Prometheus server URL |
CLUSTER_NAME |
β | default | Cluster identifier |
CLIENT_NAME |
β | default | Client/customer name |
EXCLUDED_NAMESPACES |
β | kube-system,kube-public,... | Namespaces to exclude |
REPORT_LANGUAGE |
β | spanish | Report language (spanish/english) |
JOB_POLL_INTERVAL |
β | 5 | Seconds between queue polls |
JOB_MAX_RETRIES |
β | 3 | Max retry attempts for failed jobs |
SQLITE_PATH |
β | /app/data/reports.db | SQLite database path |
LOG_LEVEL |
β | INFO | Logging level |
See .env.example for complete list.
- FastAPI Server: Runs continuously, exposing
/reportand/healthendpoints - Trigger: Can be called via HTTP POST or scheduled with Kubernetes CronJob
- AI Investigation:
- Claude receives a system prompt with available tools
- Agent autonomously decides what to investigate
- Makes iterative queries to Kubernetes and Prometheus (if available)
- Analysis: AI analyzes cluster health, resource usage, and metrics
- Report Generation: Creates HTML report, converts to PDF with WeasyPrint
- Delivery: Uploads PDF to Slack with detailed tool usage information
- Storage: Saves report to SQLite for history tracking
Claude: "Let me check the overall pod status"
β Calls: kubectl_get_pods(namespace="default", all_namespaces=True)
Claude: "I see pod X has 15 restarts. Let me investigate"
β Calls: kubectl_describe_pod(pod="X", namespace="production")
β Calls: kubectl_get_pod_logs(pod="X", namespace="production", tail=100)
Claude: "This looks like OOMKilled. Let me check memory metrics"
β Calls: prometheus_check_pod_memory(pod="X", namespace="production")
β Calls: prometheus_query(query="container_memory_working_set_bytes{pod='X'}")
Claude: "Memory usage is consistently above request. Recommending increase"
β Generates HTML report with specific recommendations
β Report includes: issue analysis, metrics charts, action plan
The system intelligently handles tool availability:
β
Kubernetes API: 5 tool types used
β’ Tools: kubectl_describe_pod, kubectl_get_deployments, kubectl_get_events, ...
β Prometheus: Connection failed
β’ Prometheus not available: All connection attempts failed
iοΈ Report generated using Kubernetes data only
Reports include:
- Executive Summary: Overall health status (π’π‘π΄)
- Top Issues: 3-5 critical problems with severity levels
- Resource Analysis: Over/under-provisioned workloads
- Prometheus Metrics: CPU, memory, disk usage (when available)
- Action Plan: Prioritized, actionable recommendations
- Footer: Generated by Watchdog AI - Helmcode
The PDF report is accompanied by a Slack message showing:
- Report generation time
- Data sources used (Kubernetes API, Prometheus)
- Tool usage statistics
- Connection status for each service
# Install dependencies pip install -e ".[dev]" # Run locally (requires kubeconfig) python -m src.main # Format code black src/ ruff check src/ # Type check mypy src/ # Build Docker image docker build -t k8s-watchdog-ai:latest .
- Read-only access: All operations are read-only (get, list, watch, describe, logs)
- RBAC: Minimal permissions required in Kubernetes
- No cluster modifications: Agent cannot modify cluster state
- Secrets management: Kubernetes secrets for sensitive data
- Connection errors: Gracefully handles unavailable services
POST /report- Generate and send report immediately (returns 202 Accepted)GET /health- Health check endpointGET /stats- Report generation statistics
- CLAUDE.md - Detailed technical documentation for AI assistants
- Architecture Overview - System design
- API Documentation - REST endpoints
Contributions are welcome! Please:
- Fork the repository
- Create a feature branch
- Make your changes
- Test thoroughly
- Submit a pull request
MIT License - see LICENSE for details.
- Anthropic Claude - AI engine
- FastAPI - Web framework
- WeasyPrint - PDF generation
- Kubernetes Python Client - K8s integration
Made with β€οΈ by Helmcode