Health

Self-hosted infrastructure health monitoring.

Notable features:

• push/pull modes of operation:
  • push: app generates an unique API endpoint and expects it to be called on regular interval (e.g. for cron jobs),
  • pull: app actively checks uptime, via:
    • http: issues a HTTP request against specified URL and verifies response status,
    • dns: checks if dns records for given domain exist,
    • ping: sends ICMP ping to specific address,
    • tcp: opens raw TCP connection (optional TLS),
    • smtp: connects and verifies EHLO response.
• tracking response times,
• tracking downtimes and error messages,
• live dashboard (with SSE/datastar),
• optional app authentication,
• alerts on failing endpoints can be delivered with:
  • email,
  • IRC,
  • XMPP,
  • matrix,
  • ntfy,
  • gotify,
  • telegram,
  • anything else via webhook or subprocess.
• status of monitored endpoints exposed as prometheus /metrics (if you wish to use prometheus-based alerting).

Configuration

Health app only requires PostgreSQL DB for data storage.

See all configuration options in compose file.

Authentication

HTTP Basic Auth

Specify env variable HEALTH_AUTH_USERS with a path to file with a list of users in htpasswd format (bcrypt only).

To generate such file:

touch health_users.txt
htpasswd -nbB admin password >> health_users.txt
htpasswd -nbB user password1 >> health_users.txt

Trust proxy

Specify env variable HEALTH_AUTH_PROXY_HEADER (e.g. Remote-User) and use an auth backend (e.g. authelia). Note, for this to be safe, app must be reachable only from reverse proxy.

Monitored endpoints

There are 2 modes how endpoints can be managed:

• via dashboard once the app is running (default)
• passed as YAML config (see example) - the app will create and delete endpoints based on provided configuration during (re)start. For this mode, set env variable HEALTH_YAML_ENDPOINTS to path to YAML file.

Alerting

Notifiers for alerting can be configured either:

• via env variables (this has limitation of having one configuration per notifier type),
• via YAML file (see example) - for this specify file path in HEALTH_YAML_NOTIFIERS (if this is provided, env variables are ignored).

Deployment

Docker compose

See example compose file.

wget https://codeberg.org/ppatrzyk/health/raw/branch/main/docker-compose.yaml
podman-compose up -d

Build

# frontend build
npm ci
npm run build
# backend build
MIX_ENV=prod mix release
# run
HEALTH_DB_HOST=localhost \
HEALTH_DB_PORT=5432 \
HEALTH_DB_NAME=health_dev \
HEALTH_DB_USER=postgres \
HEALTH_DB_PASSWORD=postgres \
_build/prod/rel/health_release/bin/health_release start

Dashboard

dashboard

Custom notifiers

There are magic strings __title__ and __message__ that will get replaced by actual alert message.

Webhook

You can configure webhook to reach any notification service that exposes HTTP api.

For instance, if you want to use 3rd-party smtp2go service, configure webhook with the following env variables:

HEALTH_WEBHOOK_URL=https://eu-api.smtp2go.com/v3/email/send
HEALTH_WEBHOOK_METHOD=POST
HEALTH_WEBHOOK_HEADERS={"X-Smtp2go-Api-Key":"<YOUR_API_KEY>","content-type":"application/json","accept":"application/json"}
HEALTH_WEBHOOK_BODY={"sender":"<SENDER_MAIL>","to":["<RECIPIENT_MAIL>"],"subject":"__title__","text_body":"__message__"}

Subprocess

You can configure subprocess to send notification via any cli command. Note, the command must return 0 exit status to be considered successful (other status code will trigger retry).

For instance, to use Simplex CLI client, configure the following:

HEALTH_SUBPROCESS_COMMAND=["/usr/local/bin/simplex-chat", "--create-bot-display-name", "health_bot", "--execute", "@recipient __message__"]

DEV

# raw socket capability set
find "$(asdf where erlang)" -type f -path "*/erts-*/bin/beam.smp"
sudo setcap cap_net_raw+ep <path>
# db
dbmate -u postgresql://postgres:postgres@localhost:5432/health_dev up
# testing endpoints
podman-compose -f "$PWD/test/docker-compose.yaml" up -d
# frontend
npm run dev
# run backend
iex -S mix
# testing
./test.sh
# run with local db
sudo podman run \
 --rm \
 -p 4000:4000 \
 -e HEALTH_DB_HOST=host.containers.internal \
 -e HEALTH_DB_PORT=5432 \
 -e HEALTH_DB_NAME=health_dev \
 -e HEALTH_DB_USER=postgres \
 -e HEALTH_DB_PASSWORD=postgres \
 --cap-add NET_RAW \
 codeberg.org/ppatrzyk/health:1.16
# build docker image
sudo podman build . -t codeberg.org/ppatrzyk/health:1.16
sudo podman push codeberg.org/ppatrzyk/health:1.16