lematt - Matt's Let's Encrypt Automation

Features

Quick Start

Installation

# Clone the repository
git clone https://github.com/mattsta/lematt
cd lematt
# Install with uv (recommended)
uv sync
# Or install with pip
pip install -e .
# Optional: Install native crypto for ~5x performance
uv sync --extra crypto
# or: pip install -e ".[crypto]"

Generate Example Configuration

# Create example TOML config (recommended for new installations)
lematt --test --init-toml
# Or use the existing INI config files in conf/

Validate Configuration

# Validate without processing any certificates
lematt --test --validate-config
# List all configured domains
lematt --test --list-domains
# Check certificate status
lematt --test --status

Run Certificate Management

# Test mode (uses Let's Encrypt staging - always test first!)
lematt --test
# Production mode
lematt --prod
# With parallel processing (up to 10 workers)
lematt --prod --parallel 5
# Dry run to see what would happen
lematt --prod --dry-run
# Force renewal of all certificates
lematt --prod --force-renew
# Process single domain
lematt --prod --domain example.com

Architecture

Design Principles

Component Overview

lematt/
├── cli.py # Command-line interface and argument parsing
├── config.py # Configuration dataclasses (LemattConfig, DomainConfig, etc.)
├── config_loader.py # Unified INI/TOML configuration loading
├── manager.py # CertificateManager - core certificate operations
├── executor.py # CertificateExecutor - parallel processing with error isolation
├── actions.py # ActionRunner - pre/post certificate workflow execution
├── crypto.py # Key generation, CSR creation, certificate parsing
├── log.py # Loguru-based structured logging
├── systemd.py # Systemd timer/service generation and installation
├── health.py # Certificate health checking and monitoring
├── notifications.py # Multi-channel alerting (email, Slack, PagerDuty, etc.)
├── display.py # Rich terminal display components (tables, progress, trees)
├── dashboard.py # Interactive live-updating dashboard
├── help.py # Contextual help system with searchable topics
└── reports.py # Comprehensive report generation (JSON, Markdown, console)

Processing Flow

1. Load Configuration (INI or TOML)
 ↓
2. Validate Configuration
 ↓
3. Load Domain List (with OCSP flags)
 ↓
4. For each domain + key type (RSA, EC):
 a. Check if renewal needed
 b. Run prepare actions (start temp servers, open ports)
 c. Generate key (if needed)
 d. Generate CSR (with OCSP Must-Staple if configured)
 e. Request certificate from Let's Encrypt
 f. Cleanup prepare actions
 ↓
5. Run post-renewal actions (upload, reload services)

Configuration

Configuration Files

TOML Configuration (Recommended)

Complete Example

# lematt.toml - Modern configuration format
[config]
# ACME challenge directory (served at /.well-known/acme-challenge/)
challenge_dir = "/var/www/html/.well-known/acme-challenge"
# Account key for Let's Encrypt
# Generate with: openssl genrsa 4096 > account.key
account_key = "/etc/lematt/account.key"
# Renew certificates this many days before expiration
reauthorize_days = 15
# RSA key size (2048, 3072, or 4096)
key_bits_rsa = 2048
# EC curve (prime256v1, secp384r1, or secp521r1)
ec_curve = "prime256v1"
# Always generate new keys on renewal (enables key rotation)
always_generate_new_keys = false
# Domain Configurations
# Each [[domains]] entry creates one certificate (RSA + EC)
[[domains]]
primary = "example.com"
sans = ["www", "mail", "api"] # Shorthand expands to www.example.com, etc.
ocsp_staple_required = false
[[domains]]
primary = "secure.example.com"
ocsp_staple_required = true # Enable OCSP Must-Staple
[[domains]]
primary = "cdn.example.com"
sans = ["static.example.com", "assets.example.com"] # Full FQDNs work too
# Action Configurations
[actions.default]
# Default actions for domains without specific overrides
upload_certs = ["rsync -avz --chmod=F644 CERTS /etc/ssl/certs/"]
upload_keys = ["rsync -avz --chmod=F640 KEYS /etc/ssl/private/"]
update = ["systemctl reload nginx"]
[actions.every]
# Actions that run for EVERY renewed certificate (in addition to others)
update = ["/usr/local/bin/backup-certs.sh"]
[actions.mail_servers]
# Override for specific domains
domains = ["mail.example.com", "smtp.example.com"]
prepare = ["ssh mail-server 'systemctl start acme-responder'"]
upload_certs = ["rsync -avz CERTS mail-server:/etc/ssl/certs/"]
upload_keys = ["rsync -avz KEYS mail-server:/etc/ssl/private/"]
update = [
 "ssh mail-server 'systemctl reload postfix'",
 "ssh mail-server 'systemctl reload dovecot'",
]

TOML Key Reference

[config] Section

[[domains]] Array

[actions.*] Tables

INI Configuration (Legacy)

lematt.conf

[config]
# Renew this many days before expiration
reauthorizeDays: 15
# OR: Renew when cert is N days old (overrides reauthorizeDays)
# generateNewCertsAfterDays: 7
# ACME challenge directory
challengeDropDir: /var/www/html/.well-known/acme-challenge
# Account key path
accountKey: /etc/lematt/account.key
# Key configuration
keyBitsRSA: 2048
curve: prime256v1
# Rotate keys on each renewal
alwaysGenerateNewKeys: no

domains

# Each line = one certificate
# Format: primary_domain [san1] [san2] ... [+ocsp|-ocsp]
# Basic single domain
example.com
# Domain with SANs (shorthand - "www" becomes "www.example.com")
example.com www mail api
# Domain with full FQDNs as SANs
example.com www.example.com mail.example.com
# Enable OCSP Must-Staple for this certificate
secure.example.com +ocsp
# Explicitly disable OCSP (default)
legacy.example.com -ocsp
# Complex example with multiple SANs and OCSP
corp.example.com intranet hr payroll git wiki +ocsp

actions.conf

# Default actions (applied when no override matches)
[default]
update: ["systemctl reload nginx"]
uploadCerts: ["rsync -avz CERTS /etc/ssl/certs/"]
uploadKeys: ["rsync -avz KEYS /etc/ssl/private/"]
# Actions for EVERY certificate (in addition to default/override)
[every]
update: ["/usr/local/bin/backup-certs.sh"]
# Override for specific domains
[mail-servers]
domains: mail.example.com smtp.example.com
prepare: ["ssh mail-server 'systemctl start acme-responder'"]
uploadCerts: ["rsync -avz CERTS mail-server:/etc/ssl/certs/"]
uploadKeys: ["rsync -avz KEYS mail-server:/etc/ssl/private/"]
update: ["ssh mail-server 'systemctl reload postfix'"]

Migration Guide: INI to TOML

Configuration Translation

Domains Translation

example.com www mail +ocsp
api.example.com

[[domains]]
primary = "example.com"
sans = ["www", "mail"]
ocsp_staple_required = true
[[domains]]
primary = "api.example.com"

Actions Translation

[default]
update: ["systemctl reload nginx"]
uploadCerts: ["rsync -avz CERTS /etc/ssl/"]
[mail-override]
domains: mail.example.com
prepare: ["ssh mail start-responder"]
update: ["ssh mail reload-postfix"]

[actions.default]
update = ["systemctl reload nginx"]
upload_certs = ["rsync -avz CERTS /etc/ssl/"]
[actions.mail_override]
domains = ["mail.example.com"]
prepare = ["ssh mail start-responder"]
update = ["ssh mail reload-postfix"]

Key Differences

CLI Reference

Mode Selection (Required)

lematt --test # Use Let's Encrypt staging endpoint (always test first!)
lematt --prod # Use Let's Encrypt production endpoint

Information Commands

# Show certificate status
lematt --test --status
# Status as JSON (for monitoring systems)
lematt --test --status --json
# List all configured domains
lematt --test --list-domains
# List domains as JSON
lematt --test --list-domains --json
# Validate configuration only
lematt --test --validate-config

Processing Options

# Parallel processing (max 10 workers)
lematt --prod --parallel 5
# Process single domain
lematt --prod --domain example.com
# Force renewal regardless of expiration
lematt --prod --force-renew
# Dry run (show what would happen)
lematt --prod --dry-run
# Minimize output (for cron)
lematt --prod --cron

Configuration

# Use custom config path
lematt --test --config /path/to/lematt.conf
# Generate example TOML config
lematt --test --init-toml
# Verbose output
lematt --test -v

Systemd Commands

# Install systemd timer
sudo lematt --prod --install-systemd
# Install with preset schedule
sudo lematt --prod --install-systemd --systemd-preset aggressive
# Check timer status
lematt --prod --systemd-status
lematt --prod --systemd-status --json
# Uninstall
sudo lematt --prod --uninstall-systemd

Health Check Commands

# Basic health check
lematt --test --health-check
# JSON output
lematt --test --health-check --json
# Prometheus metrics
lematt --test --health-check --prometheus
# Check live certificates (TLS connection)
lematt --test --health-check --check-live
# Custom thresholds
lematt --test --health-check --warning-days 21 --critical-days 14
# Write to file for monitoring
lematt --test --health-check --write-health /var/lib/lematt/health.json

Notification Commands

# Create notification config
sudo lematt --prod --init-notifications
# Test notifications
lematt --test --test-notification

Complete Options

usage: lematt [-h] (--prod | --test) [--cron] [--parallel N] [--config PATH]
 [-v] [--dry-run] [--status] [--json] [--list-domains]
 [--validate-config] [--domain DOMAIN] [--force-renew]
 [--init-toml] [--install-systemd] [--uninstall-systemd]
 [--systemd-status] [--systemd-preset PRESET]
 [--health-check] [--check-live] [--warning-days N]
 [--critical-days N] [--prometheus] [--write-health PATH]
 [--init-notifications] [--test-notification]
 [--dashboard] [--dashboard-refresh SECONDS] [--dashboard-compact]
 [--report] [--report-output PATH] [--report-format FORMAT]
 [--help-topic [TOPIC]] [--help-search QUERY]
Mode:
 --prod Use production Let's Encrypt endpoint
 --test Use staging Let's Encrypt endpoint
Processing:
 --cron Minimize output (only errors and renewals)
 --parallel N Process N certificates in parallel (max 10)
 --config PATH Path to configuration file
 -v, --verbose Enable debug output
 --dry-run Show what would be done without changes
 --domain DOMAIN Process only specified domain
 --force-renew Force renewal regardless of expiration
Information:
 --status Show certificate status and exit
 --json Output as JSON
 --list-domains List all configured domains and exit
 --validate-config Validate configuration without processing
 --init-toml Create example lematt.toml and exit
Systemd Automation:
 --install-systemd Install systemd timer and service
 --uninstall-systemd Remove systemd timer and service
 --systemd-status Show status of systemd timer
 --systemd-preset Timer schedule: default, aggressive, conservative, weekly
Health Checks:
 --health-check Run health check on all certificates
 --check-live Also check live certificates served by domains
 --warning-days N Days before expiry to warn (default: 14)
 --critical-days N Days before expiry for critical (default: 7)
 --prometheus Output health check as Prometheus metrics
 --write-health PATH Write health status to file
Notifications:
 --init-notifications Create example notification config
 --test-notification Send a test notification
Interactive UI:
 --dashboard Launch interactive certificate dashboard
 --dashboard-refresh Dashboard refresh interval in seconds (default: 5)
 --dashboard-compact Use compact display mode for dashboard
 --report Generate comprehensive certificate report
 --report-output Save report to file (auto-detects format)
 --report-format Report format: console, json, markdown
 --help-topic Show detailed help for a topic (list all if no arg)
 --help-search Search help topics for a keyword

JSON Output Examples

Certificate Status

lematt --test --status --json

{
 "certificates": [
 {
 "domain": "example.com",
 "san_domains": ["www.example.com"],
 "key_type": "rsa",
 "cert_path": "/path/to/cert.pem",
 "exists": true,
 "status": "ok",
 "expires": "2024年06月15日T12:00:00",
 "days_until_expiry": 45
 },
 {
 "domain": "example.com",
 "san_domains": ["www.example.com"],
 "key_type": "ec",
 "cert_path": "/path/to/cert.pem",
 "exists": true,
 "status": "renew_soon",
 "expires": "2024年05月10日T12:00:00",
 "days_until_expiry": 12
 }
 ],
 "summary": {
 "total": 4,
 "ok": 2,
 "renew_soon": 1,
 "critical": 0,
 "expired": 0,
 "missing": 1
 },
 "using_native_crypto": true
}

Domain List

lematt --test --list-domains --json

[
 {
 "primary": "example.com",
 "sans": ["www.example.com", "mail.example.com"],
 "all_domains": ["example.com", "www.example.com", "mail.example.com"],
 "ocsp_staple_required": false
 },
 {
 "primary": "secure.example.com",
 "sans": [],
 "all_domains": ["secure.example.com"],
 "ocsp_staple_required": true
 }
]

Workflow Actions

Action Types

Variable Expansion

Execution Order

Action Groups

OCSP Must-Staple

Per-Domain Configuration

example.com www +ocsp # Enable OCSP
legacy.example.com -ocsp # Explicitly disable (default)

[[domains]]
primary = "example.com"
sans = ["www"]
ocsp_staple_required = true

Server Configuration Required

ssl_stapling on;
ssl_stapling_verify on;
resolver 8.8.8.8 8.8.4.4 valid=300s;
resolver_timeout 5s;

SSLUseStapling On
SSLStaplingCache shmcb:${APACHE_RUN_DIR}/ssl_stapling(32768)

Performance

Native Cryptography

uv sync --extra crypto
# or: pip install cryptography

lematt --test --status # Shows "(Using native cryptography library)"

Parallel Processing

# Process 5 certificates in parallel
lematt --prod --parallel 5

Testing

Always Test First

# 1. Validate configuration
lematt --test --validate-config
# 2. Check domain list
lematt --test --list-domains
# 3. Dry run
lematt --test --dry-run
# 4. Full test run (uses staging endpoint)
lematt --test
# 5. Check results
lematt --test --status

Test vs Production

Automated Renewals with systemd

Quick Setup

# Install systemd timer (default: twice daily)
sudo lematt --prod --install-systemd
# Check timer status
lematt --prod --systemd-status
systemctl status lematt-renew.timer
# View logs
journalctl -u lematt-renew.service -f

Timer Presets

# Install with preset
sudo lematt --prod --install-systemd --systemd-preset aggressive
# Uninstall
sudo lematt --prod --uninstall-systemd

Systemd Unit Details

Generated Service Unit

[Unit]
Description=Lematt Certificate Renewal Service
After=network-online.target
[Service]
Type=oneshot
User=root
ExecStart=/usr/bin/env python3 -m lematt --config /etc/lematt/lematt.conf
# Security hardening
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
NoNewPrivileges=true
ReadWritePaths=/etc/lematt
MemoryMax=512M
CPUQuota=50%
[Install]
WantedBy=multi-user.target

Manual Trigger

# Run immediately (outside scheduled time)
sudo systemctl start lematt-renew.service
# Follow output
sudo journalctl -u lematt-renew.service -f

Health Checks and Monitoring

Basic Health Check

# Check all certificates
lematt --test --health-check
# Output:
# Health Check: ✓ healthy
# Certificates: 0 critical, 0 warning, 4 healthy
# ------------------------------------------------------------
# ✓ example.com (rsa): Certificate valid for 45 days
# ✓ example.com (ec): Certificate valid for 45 days
# ⚠ other.com (rsa): Certificate expires in 12 days
# ✗ expired.com (ec): Certificate EXPIRED 5 days ago

Health Check Options

# JSON output for monitoring systems
lematt --test --health-check --json
# Prometheus metrics format
lematt --test --health-check --prometheus
# Write health status to file (for external monitoring)
lematt --test --health-check --write-health /var/lib/lematt/health.json
# Check live certificates served by domains
lematt --test --health-check --check-live
# Custom warning/critical thresholds
lematt --test --health-check --warning-days 21 --critical-days 14

Exit Codes

JSON Health Output

{
 "status": "warning",
 "summary": "Certificates: 1 critical, 2 warning, 8 healthy",
 "checked_at": "2024年05月01日T12:00:00",
 "counts": {
 "healthy": 8,
 "warning": 2,
 "critical": 1,
 "unknown": 0,
 "total": 11
 },
 "certificates": [
 {
 "domain": "example.com",
 "key_type": "rsa",
 "status": "healthy",
 "message": "Certificate valid for 45 days",
 "cert_path": "/etc/lematt/prod/cert/example.com-cert.pem",
 "days_until_expiry": 45,
 "not_after": "2024年06月15日T00:00:00",
 "issuer": "CN=Let's Encrypt Authority X3"
 }
 ]
}

Prometheus Metrics

lematt --test --health-check --prometheus

# HELP lematt_certificate_expiry_days Days until certificate expires
# TYPE lematt_certificate_expiry_days gauge
lematt_certificate_expiry_days{domain="example.com",key_type="rsa"} 45
lematt_certificate_expiry_days{domain="example.com",key_type="ec"} 45
# HELP lematt_certificate_status Certificate health status (0=healthy, 1=warning, 2=critical, 3=unknown)
# TYPE lematt_certificate_status gauge
lematt_certificate_status{domain="example.com",key_type="rsa"} 0
lematt_certificate_status{domain="example.com",key_type="ec"} 0
# HELP lematt_certificates_total Total number of certificates by status
# TYPE lematt_certificates_total gauge
lematt_certificates_total{status="healthy"} 4
lematt_certificates_total{status="warning"} 0
lematt_certificates_total{status="critical"} 0

Monitoring Integration

# /etc/cron.d/lematt-health
*/30 * * * * root lematt --prod --health-check --write-health /var/lib/lematt/health.json

#!/bin/bash
# /usr/local/lib/nagios/plugins/check_lematt
lematt --prod --health-check 2>/dev/null
exit $?

Interactive Dashboard

Launch Dashboard

# Launch interactive dashboard
lematt --test --dashboard
# Custom refresh interval (default: 5 seconds)
lematt --test --dashboard --dashboard-refresh 10
# Compact display mode
lematt --test --dashboard --dashboard-compact

Dashboard Features

Keyboard Controls

Dashboard Views

Summary Reports

Generate Reports

# Display report in terminal
lematt --test --report
# Save as JSON
lematt --test --report --report-output report.json
# Save as Markdown
lematt --test --report --report-output report.md
# Specify format explicitly
lematt --test --report --report-format markdown --report-output status.md

Report Contents

Report Formats

Renewal Schedule Priorities

Contextual Help System

View Help Topics

# List all help topics
lematt --test --help-topic
# View specific topic
lematt --test --help-topic run
lematt --test --help-topic dashboard
lematt --test --help-topic configuration
lematt --test --help-topic systemd
lematt --test --help-topic troubleshooting
# Search help topics
lematt --test --help-search certificate
lematt --test --help-search notification

Available Help Topics

Help Categories

Rich Terminal Display

Features

Display Components

from lematt import (
 console,
 print_banner,
 print_success,
 print_warning,
 print_error,
 print_info,
 create_certificate_table,
 create_health_summary,
 create_domain_tree,
)
# Print status messages
print_success("Certificate renewed successfully")
print_warning("Certificate expires in 10 days")
print_error("Failed to renew certificate")
print_info("Checking certificate status...")
# Create display components
table = create_certificate_table(certificates)
console.print(table)
tree = create_domain_tree(domains)
console.print(tree)

Status Styles

Alerting and Notifications

Configure Notifications

[notifications]
# Email (requires sendmail or SMTP)
email_to = "admin@example.com"
email_from = "lematt@example.com"
# Slack webhook
webhook_url = "https://hooks.slack.com/services/XXX/YYY/ZZZ"
webhook_format = "slack" # slack, discord, or generic
# PagerDuty (only alerts on failures)
pagerduty_key = "your-integration-key"
# Ntfy push notifications (https://ntfy.sh)
ntfy_topic = "my-cert-alerts"
ntfy_server = "https://ntfy.sh"
# Custom command
custom_command = "/usr/local/bin/my-notify-script.sh"
# Journald logging (enabled by default)
journald_enabled = true
# When to notify
notify_on_failure = true
notify_on_warning = true
notify_on_success = false

Test Notifications

# Send test notification to verify configuration
lematt --test --test-notification
# Output:
# Testing 3 notification backend(s)...
# ✓ email
# ✓ webhook-slack
# ✓ journald
# All notifications sent successfully: 3

Notification Backends

Shell Script Notifications

# /etc/lematt/notify.conf
NOTIFY_EMAIL="admin@example.com"
NOTIFY_WEBHOOK="https://hooks.slack.com/services/XXX/YYY/ZZZ"
PAGERDUTY_KEY="your-integration-key"
NTFY_TOPIC="my-cert-alerts"
NOTIFY_CUSTOM_CMD="/usr/local/bin/my-script.sh"

Notification Events

Slack Notification Example

{
 "attachments": [
 {
 "color": "danger",
 "title": "Certificate Renewal Failed",
 "text": "Failed to renew 2 certificates",
 "fields": [
 { "title": "Host", "value": "web-server-01", "short": true },
 { "title": "Time", "value": "2024年05月01日 12:00:00", "short": true },
 {
 "title": "Domains",
 "value": "example.com, api.example.com",
 "short": true
 }
 ],
 "footer": "Lematt Certificate Manager"
 }
 ]
}

Custom Notification Script

#!/bin/bash
# /usr/local/bin/my-notify-script.sh
# Receives: event_type title message hostname timestamp
EVENT_TYPE="1ドル"
TITLE="2ドル"
MESSAGE="3ドル"
HOSTNAME="4ドル"
TIMESTAMP="5ドル"
# Your custom notification logic here
curl -X POST "https://your-endpoint.com/webhook" \
 -H "Content-Type: application/json" \
 -d "{\"event\": \"$EVENT_TYPE\", \"title\": \"$TITLE\", \"message\": \"$MESSAGE\"}"

Troubleshooting

Common Issues

# Ensure config exists
ls conf/lematt.conf conf/lematt.toml
# Or specify path
lematt --test --config /path/to/lematt.conf

# Generate account key
openssl genrsa 4096 > /etc/lematt/account.key
chmod 600 /etc/lematt/account.key

mkdir -p /var/www/html/.well-known/acme-challenge
# Ensure web server serves this path

Logging

lematt --test -v

Programmatic Usage

Python API

from lematt import (
 LemattConfig,
 DomainConfig,
 CertificateManager,
 CertificateExecutor,
 ActionRunner,
 DomainActions,
)
# Create configuration
config = LemattConfig(
 config_base="/etc/lematt",
 challenge_dir="/var/www/challenges",
 account_key="/etc/lematt/account.key",
 is_test=True,
)
# Define domains
domains = [
 DomainConfig(
 primary_domain="example.com",
 san_domains=["www.example.com"],
 ocsp_staple_required=False,
 ),
]
# Load actions
actions = ActionRunner(config)
actions.load_actions()
# Process certificates
executor = CertificateExecutor(
 config=config,
 max_workers=5,
 rate_limit=10.0,
)
summary = executor.process_batch(
 domains=domains,
 domain_actions=actions.domain_actions,
)
print(f"Renewed: {summary.renewed_count}, Failed: {summary.failed_count}")

Using ConfigLoader

from pathlib import Path
from lematt import ConfigLoader
# Load configuration (auto-detects TOML or INI)
loader = ConfigLoader(config_base=Path("/etc/lematt"))
loader.load()
# Get typed configuration
config = loader.get_lematt_config(
 is_test=True,
 concurrency=5,
)
# Get domains and actions
domains = loader.get_domains()
actions = loader.get_actions()

Security Considerations

Contributing

Development Setup

git clone https://github.com/mattsta/lematt
cd lematt
uv sync --dev
uv run ruff check src/
uv run pytest

License

Acknowledgments