A self-healing system that detects production incidents, decides the correct remediation action, executes it automatically, generates an AI-powered RCA report with full incident timeline, and opens a Jira ticket — with zero human intervention.
Prometheus Alert → Alertmanager Webhook → API → Container Logs
↓
AI Triage (log-triage-service)
↓
Condition Engine → Decision
↓
Action Executed
↓
Jira Ticket + RCA Report + Timeline Auto-Generated
Real example:
- Prometheus detects CPU_SPIKE or OOM_KILL
- Alertmanager sends webhook to
/incident - Container logs pulled automatically
- Logs sent to log-triage-service — AI generates root cause hypothesis + severity
- Condition engine checks restart history from SQLite
- If restart_count < 3 → restarts container; if >= 3 → escalates
- Jira ticket auto-created with AI root cause and next steps
- Full RCA report saved as JSON — includes incident timeline, AI analysis, and Jira ticket key
- Condition-based decision engine — evaluates restart history, applies escalation logic
- AI-powered diagnosis — integrates with log-triage-service to analyze container logs and generate root cause hypothesis with severity score
- Incident timeline — every RCA report includes full history of past actions for that container, across sessions
- Cooldown protection — prevents action spam on repeated alerts (configurable, default 30s)
- Persistent state (SQLite) — restart count and incident history survive service restarts
- Auto-generated RCA reports — structured JSON per incident: timestamp, condition matched, action taken, AI analysis, timeline, Jira ticket key
- Jira integration — incident ticket auto-created with AI root cause and next steps
- HTTP API — accepts both Alertmanager webhook format and manual POST requests
- Real metrics via cAdvisor — actual container CPU and memory data, not synthetic
┌─────────────┐ ┌──────────────┐ ┌─────────────────┐
│ Prometheus │────▶│ Alertmanager │────▶│ Flask API │
└─────────────┘ └──────────────┘ └────────┬────────┘
│
┌───────────▼───────────┐
│ Get Container Logs │
└───────────┬───────────┘
│
┌───────────▼───────────┐
│ log-triage-service │
│ (AI Root Cause + Sev) │
└───────────┬───────────┘
│
┌────────▼────────┐
│ Condition Engine │
└────────┬────────┘
│
┌──────────────┼──────────────┐
│ │ │
┌──────▼─────┐ ┌─────▼──────┐ ┌────▼──────┐
│ Restart │ │ Scale │ │ Escalate │
└──────┬─────┘ └─────┬──────┘ └────┬──────┘
│ │ │
└──────────────┼──────────────┘
│
┌──────────────┼──────────────┐
│ │
┌──────▼──────┐ ┌───────▼──────┐
│ JSON Report │ │ Jira Ticket │
│ + Timeline │ │ (AUTO) │
│ + SQLite │ └──────────────┘
└─────────────┘
| Incident | Condition | Action |
|---|---|---|
| OOM_KILL | restart_count < 3 | restart_container |
| OOM_KILL | restart_count >= 3 | escalate + Jira ticket |
| CRASH_LOOP | restart_count < 3 | restart_container |
| CRASH_LOOP | restart_count >= 3 | escalate + Jira ticket |
| CPU_SPIKE | — | scale_container |
{
"timestamp": "2026年04月12日T21:08:47",
"incident_type": "CPU_SPIKE",
"container": "my-app",
"restart_count": 3,
"condition_matched": "none",
"action_taken": "scale_container",
"result": "success",
"timeline": [
{ "time": "2026年04月11日T14:39:35", "event": "OOM_KILL → escalate" },
{ "time": "2026年04月11日T14:40:49", "event": "OOM_KILL → escalate" },
{ "time": "2026年04月11日T14:45:26", "event": "OOM_KILL → escalate" },
{ "time": "2026年04月12日T21:08:47", "event": "current → scale_container" }
],
"ai_analysis": {
"severity": "LOW",
"root_cause": "No known critical patterns detected in the provided log snippet.",
"next_steps": [
"Provide a longer log window around the error including stack trace.",
"Share timestamp, request id/correlation id, and environment info."
],
"ticket_title": "Incident: Log analysis result (LOW)"
},
"jira_ticket": "RE-2"
}- Python — condition engine, action executor, report generator, timeline builder
- Flask — webhook API
- SQLite — persistent incident state and history
- log-triage-service — AI-powered log analysis, root cause hypothesis, severity scoring (Java 17 + Spring Boot + Hugging Face LLM)
- Prometheus + Alertmanager — alert generation and routing
- cAdvisor — real container CPU/memory metrics
- Grafana — live infrastructure dashboard
- Jira API — auto-generated incident tickets
- Docker + Docker Compose — container management and test environment
- YAML — runbook definitions (human-readable, easily extendable)
# Start infrastructure docker compose up -d # Start log-triage-service (see https://github.com/kadak25/log-triage-service) cd ../log-triage-service && docker compose up -d # Start API cd ../runbook-executor set JIRA_TOKEN=your_token_here # Windows export JIRA_TOKEN=your_token_here # Linux/Mac python api.py # Manual test curl -X POST http://localhost:5000/incident \ -H "Content-Type: application/json" \ -d '{"incident_type": "OOM_KILL", "context": {"container": "my-app"}}'
docker run --name cpu-stress --rm progrium/stress --cpu 4 --timeout 60
Watch Prometheus fire the alert, Alertmanager route it to the API, AI analyze the logs, and the system scale + open a Jira ticket automatically.
L1 support teams spend significant time on repetitive incident response:
- Wake up at 3am
- Check runbook
- Analyze logs manually
- Restart container
- Write Jira ticket
This system eliminates that entire loop. The runbook is the code. The logs are analyzed automatically. The ticket writes itself. The timeline builds itself.
Add a new incident type in 3 lines of YAML:
DISK_FULL: actions: - rotate_logs
Then implement rotate_logs() in executor.py. Done.
- log-triage-service — AI-powered log analysis tool used as the diagnosis layer in this system