Infrastructure Management Checklist #312

ryansquared commented 2026年06月19日 11:19:57 +02:00

This is a living list of required infrastructure upgrades:

Standalone Components:

• ECR repositories for enclave components (execd, locksmith, bootproof, etc.)
• ECR repositories for platform components (api, email, metering, etc.)
• S3 bucket for managing database backups
• RDS instance, if we confirm we don't want it running in-cluster
• Forgejo Actions runner machine and CI system, to build said components on-push
  • Runner machine must have git verify-commit baked in. Might require a custom version of the runner.

Infrastructure should be deployed using Tofu with either an in-repository encrypted state, or a one-time ClickOps'd bucket. Non-cluster machines should be running Debian with Ansible configurations. Future upgrades to stagex-based single-use images can be considered later.

Cluster Components

• Generic Kubernetes necessities, such as cert-manager, traefik ingress, external DNS, and whatever else is necessary
• Mimir, Loki, etc. Grafana monitoring stack, with dashboards and alerts monitoring:
  • HTTP 4xx/5xx rates
  • Persistent volume usage
  • Amount of deployed resources versus capacity (we'd need a custom metric for this, I imagine)
  • In-cluster failed compute resources
• Platform components running in-cluster
  • No more make up on the boxen!
• CronJob in the cluster, to automate database backups to aforementioned offsite S3 bucket

Applications should be deployed using Kustomize into their own namespaces. Secrets should be managed using ksops. Follow guidelines in https://git.distrust.co/public/stack for a reasonable base to get started, but we should also start building out kustomizations to be reusable for separate environments, so we can each have our own testing infrastructure without stepping on each others' toes.

---

Edit as necessary to add further components.

This is a living list of required infrastructure upgrades: ## Standalone Components: * ECR repositories for enclave components (execd, locksmith, bootproof, etc.) * ECR repositories for platform components (api, email, metering, etc.) * S3 bucket for managing database backups * RDS instance, if we confirm we don't want it running in-cluster * Forgejo Actions runner machine and CI system, to build said components on-push * Runner machine must have `git verify-commit` baked in. Might require a custom version of the runner. Infrastructure should be deployed using Tofu with either an in-repository encrypted state, or a one-time ClickOps'd bucket. Non-cluster machines should be running Debian with Ansible configurations. Future upgrades to stagex-based single-use images can be considered later. ## Cluster Components * Generic Kubernetes necessities, such as cert-manager, traefik ingress, external DNS, and whatever else is necessary * Mimir, Loki, etc. Grafana monitoring stack, with dashboards and alerts monitoring: * HTTP 4xx/5xx rates * Persistent volume usage * Amount of deployed resources versus capacity (we'd need a custom metric for this, I imagine) * In-cluster failed compute resources * Platform components running in-cluster * No more `make up` on the boxen! * CronJob in the cluster, to automate database backups to aforementioned offsite S3 bucket Applications should be deployed using Kustomize into their own namespaces. Secrets should be managed using ksops. Follow guidelines in https://git.distrust.co/public/stack for a _reasonable_ base to get started, but we should also start building out kustomizations to be reusable for separate environments, so we can each have our own testing infrastructure without stepping on each others' toes. --- Edit as necessary to add further components.

vkobel commented 2026年06月25日 15:23:37 +02:00

• ECR pull-through cache for stagex/* so base layers come from inside AWS, not Docker Hub over www

- ECR pull-through cache for stagex/* so base layers come from inside AWS, not Docker Hub over www