Add opt-in Sentry error reporting for platform services and dashboard #377

caution-bot commented 2026年07月10日 22:16:48 +02:00

Summary

Add opt-in Sentry error reporting for the operator-facing Caution platform services and dashboard, with separate development and production Sentry environments. The integration should improve crash/error visibility without weakening Caution's confidentiality, attestation, reproducibility, or secret-handling guarantees.

Proposed implementation

Backend Rust services

Add a small shared workspace crate, for example src/observability / caution-observability, that centralizes Sentry and tracing setup for:

• src/api
• src/gateway
• src/metering
• src/email-service

The crate should:

• initialize Sentry only when SENTRY_DSN is set and non-empty;
• keep existing stdout tracing behavior intact;
• integrate with the existing tracing usage via sentry + sentry-tracing;
• capture panics and tracing::error! events;
• treat tracing::warn! events as breadcrumbs unless there is a specific reason to promote one to an event;
• attach service name, environment, release, and request ID/correlation ID where safe;
• scrub sensitive headers and values before anything can leave the process;
• never capture request or response bodies by default.

Recommended Rust crates, subject to compatibility with the repository's pinned toolchain/container builds:

sentry = { version = "0.48.4", default-features = false, features = [
 "backtrace",
 "contexts",
 "panic",
 "tracing",
 "tower",
 "tower-http",
 "tower-axum-matched-path",
 "reqwest",
 "rustls",
] }
sentry-tracing = "0.48.4"

Use the rustls path rather than native TLS where feasible because these services are built for static/musl StageX-style runtime images.

Axum request context

Layer Sentry into the Axum routers where appropriate, but prefer route templates/matched paths over raw user-controlled URLs. Include safe context such as:

• service name (api, gateway, metering, email-service);
• release/git SHA;
• environment (development, production, etc.);
• HTTP method and matched route;
• status code;
• sanitized request ID.

Do not capture request/response bodies.

The gateway already has X-Request-Id middleware. Before using this value as a Sentry tag/context, validate it strictly, e.g. accept UUIDs only and generate a new UUID otherwise, to avoid log/header/tag injection from user-supplied request IDs.

Frontend Vue dashboard

Add an opt-in Vue integration in frontend/src/main.js using @sentry/vue:

• initialize only when VITE_SENTRY_DSN is set;
• set sendDefaultPii: false;
• keep tracing sample rate at 0 initially;
• do not enable Session Replay in the first pass because the dashboard handles WebAuthn, SSH keys, billing, attestation, and deployment state.

Add build-time variables to the frontend build path used by containerfiles/Containerfile.gateway and containerfiles/Containerfile.frontend.

Source map upload, if desired, should be a follow-up in CI/release automation using a CI-only SENTRY_AUTH_TOKEN. Do not pass SENTRY_AUTH_TOKEN through Docker build args, final images, env.example, systemd units, or runtime .env files.

Sentry setup instructions

1. Create Sentry organization/projects

In Sentry, create separate projects for backend and frontend so SDK settings and alert rules can differ:

• Backend project: caution-platform-backend, platform Rust
• Frontend project: caution-platform-frontend, platform JavaScript/Vue

Use separate Sentry environments instead of separate projects for dev vs prod unless there is a compliance or access-control reason to fully isolate them:

• development
• production

If staging is used, also add:

• staging

2. Development environment

Development should be opt-in and low/no-noise by default.

Recommended local .env additions:

# Observability / Sentry (optional)
SENTRY_DSN=https://<backend-public-key>@<sentry-host>/<backend-project-id>
SENTRY_ENVIRONMENT=development
SENTRY_RELEASE=dev-local
SENTRY_TRACES_SAMPLE_RATE=0
SENTRY_PROFILES_SAMPLE_RATE=0
SENTRY_SEND_DEFAULT_PII=false
# Frontend Sentry (optional; public DSN is safe to expose, auth tokens are not)
VITE_SENTRY_DSN=https://<frontend-public-key>@<sentry-host>/<frontend-project-id>
VITE_SENTRY_ENVIRONMENT=development
VITE_SENTRY_RELEASE=dev-local
VITE_SENTRY_TRACES_SAMPLE_RATE=0

For ordinary local development, leave both DSNs empty to disable Sentry entirely. Developers can set them only when intentionally testing observability.

Development verification steps:

1. Run the affected service with SENTRY_DSN set.
2. Trigger a controlled non-secret test error or panic in a development-only path or test harness.
3. Verify the event appears in Sentry with environment development.
4. Confirm no cookies, authorization headers, internal service secrets, request bodies, encrypted secret material, raw attestation documents, cloud credentials, SMTP credentials, or Paddle secrets appear in the event.
5. Clear the test hook before merging unless it is test-only and guarded.

3. Production environment

Production should use real release identifiers and conservative sampling.

Recommended production runtime .env additions:

SENTRY_DSN=https://<backend-public-key>@<sentry-host>/<backend-project-id>
SENTRY_ENVIRONMENT=production
SENTRY_RELEASE=<git-sha-or-release-version>
SENTRY_TRACES_SAMPLE_RATE=0
SENTRY_PROFILES_SAMPLE_RATE=0
SENTRY_SEND_DEFAULT_PII=false
VITE_SENTRY_DSN=https://<frontend-public-key>@<sentry-host>/<frontend-project-id>
VITE_SENTRY_ENVIRONMENT=production
VITE_SENTRY_RELEASE=<git-sha-or-release-version>
VITE_SENTRY_TRACES_SAMPLE_RATE=0

Production rollout guidance:

1. Deploy backend Sentry first with SENTRY_TRACES_SAMPLE_RATE=0 and frontend tracing disabled.
2. Verify events are grouped by service and release.
3. Verify sensitive data scrubbing against real-shaped but fake test requests.
4. Only after error capture is stable, consider enabling a very low traces sample rate such as 0.01, and only for routes where privacy and volume are understood.
5. Keep Session Replay disabled unless a separate privacy/security review approves it.
6. Set Sentry alert routing for production only; keep development alerts muted or routed to a low-priority channel.

4. Source maps and releases

Frontend source maps should be uploaded from CI/release automation, not from runtime containers.

If source map upload is added later:

• use @sentry/vite-plugin or sentry-cli in CI;
• store SENTRY_AUTH_TOKEN only as a CI secret;
• associate source maps with VITE_SENTRY_RELEASE;
• delete or avoid publishing source maps in served static assets if that is the desired production policy;
• do not commit, print, or bake SENTRY_AUTH_TOKEN into images.

Security and privacy requirements

Sentry must not send or store:

• request bodies or response bodies;
• cookies;
• Authorization headers;
• INTERNAL_SERVICE_SECRET;
• Paddle API keys, webhook secrets, checkout payload secrets, or billing provider secrets;
• SMTP credentials;
• cloud credentials;
• plaintext .env values;
• encrypted secret material from .caution/secrets/*.asc;
• Locksmith shard contents or private keys;
• raw attestation documents, nonce material, PCR blobs, or certificate chains unless explicitly reviewed and redacted;
• customer workload secrets or deployment-time secrets;
• private SSH keys, WebAuthn challenge responses, or passkey credential material.

Do not initialize Sentry inside enclave/customer workload code or deterministic/reproducible build paths. Sentry should be limited to operator-facing platform services and the dashboard.

Acceptance criteria

• SENTRY_DSN unset or empty means backend Sentry is fully disabled and services behave as they do today.
• VITE_SENTRY_DSN unset or empty means frontend Sentry is fully disabled.
• Backend services emit Sentry events for panics and error-level tracing events when enabled.
• Events include service, environment, and release context.
• Request context uses matched routes/sanitized IDs rather than raw attacker-controlled values where possible.
• Sensitive headers and known secret-bearing fields are scrubbed before events are sent.
• Request/response bodies are not captured by default.
• Frontend captures uncaught Vue/browser errors when enabled, with sendDefaultPii: false and replay disabled.
• env.example, deployment docs, or relevant README sections document dev and prod Sentry setup.
• Tests cover configuration parsing/defaults and scrubbing behavior.
• Existing regular checks continue to pass.

Suggested test plan

Backend:

cargo check -p api -p gateway -p metering -p email-service
cargo test -p caution-observability
cargo test --workspace

Frontend:

cd frontend
npm install
npm run build

Container/build smoke tests where feasible:

make build-api
make build-gateway
make build-metering
make build-email

## Summary Add opt-in Sentry error reporting for the operator-facing Caution platform services and dashboard, with separate development and production Sentry environments. The integration should improve crash/error visibility without weakening Caution's confidentiality, attestation, reproducibility, or secret-handling guarantees. ## Proposed implementation ### Backend Rust services Add a small shared workspace crate, for example `src/observability` / `caution-observability`, that centralizes Sentry and tracing setup for: - `src/api` - `src/gateway` - `src/metering` - `src/email-service` The crate should: - initialize Sentry only when `SENTRY_DSN` is set and non-empty; - keep existing stdout tracing behavior intact; - integrate with the existing `tracing` usage via `sentry` + `sentry-tracing`; - capture panics and `tracing::error!` events; - treat `tracing::warn!` events as breadcrumbs unless there is a specific reason to promote one to an event; - attach service name, environment, release, and request ID/correlation ID where safe; - scrub sensitive headers and values before anything can leave the process; - never capture request or response bodies by default. Recommended Rust crates, subject to compatibility with the repository's pinned toolchain/container builds: ```toml sentry = { version = "0.48.4", default-features = false, features = [ "backtrace", "contexts", "panic", "tracing", "tower", "tower-http", "tower-axum-matched-path", "reqwest", "rustls", ] } sentry-tracing = "0.48.4" ``` Use the rustls path rather than native TLS where feasible because these services are built for static/musl StageX-style runtime images. ### Axum request context Layer Sentry into the Axum routers where appropriate, but prefer route templates/matched paths over raw user-controlled URLs. Include safe context such as: - service name (`api`, `gateway`, `metering`, `email-service`); - release/git SHA; - environment (`development`, `production`, etc.); - HTTP method and matched route; - status code; - sanitized request ID. Do not capture request/response bodies. The gateway already has `X-Request-Id` middleware. Before using this value as a Sentry tag/context, validate it strictly, e.g. accept UUIDs only and generate a new UUID otherwise, to avoid log/header/tag injection from user-supplied request IDs. ### Frontend Vue dashboard Add an opt-in Vue integration in `frontend/src/main.js` using `@sentry/vue`: - initialize only when `VITE_SENTRY_DSN` is set; - set `sendDefaultPii: false`; - keep tracing sample rate at `0` initially; - do not enable Session Replay in the first pass because the dashboard handles WebAuthn, SSH keys, billing, attestation, and deployment state. Add build-time variables to the frontend build path used by `containerfiles/Containerfile.gateway` and `containerfiles/Containerfile.frontend`. Source map upload, if desired, should be a follow-up in CI/release automation using a CI-only `SENTRY_AUTH_TOKEN`. Do not pass `SENTRY_AUTH_TOKEN` through Docker build args, final images, `env.example`, systemd units, or runtime `.env` files. ## Sentry setup instructions ### 1. Create Sentry organization/projects In Sentry, create separate projects for backend and frontend so SDK settings and alert rules can differ: - Backend project: `caution-platform-backend`, platform Rust - Frontend project: `caution-platform-frontend`, platform JavaScript/Vue Use separate Sentry environments instead of separate projects for dev vs prod unless there is a compliance or access-control reason to fully isolate them: - `development` - `production` If staging is used, also add: - `staging` ### 2. Development environment Development should be opt-in and low/no-noise by default. Recommended local `.env` additions: ```dotenv # Observability / Sentry (optional) SENTRY_DSN=https://<backend-public-key>@<sentry-host>/<backend-project-id> SENTRY_ENVIRONMENT=development SENTRY_RELEASE=dev-local SENTRY_TRACES_SAMPLE_RATE=0 SENTRY_PROFILES_SAMPLE_RATE=0 SENTRY_SEND_DEFAULT_PII=false # Frontend Sentry (optional; public DSN is safe to expose, auth tokens are not) VITE_SENTRY_DSN=https://<frontend-public-key>@<sentry-host>/<frontend-project-id> VITE_SENTRY_ENVIRONMENT=development VITE_SENTRY_RELEASE=dev-local VITE_SENTRY_TRACES_SAMPLE_RATE=0 ``` For ordinary local development, leave both DSNs empty to disable Sentry entirely. Developers can set them only when intentionally testing observability. Development verification steps: 1. Run the affected service with `SENTRY_DSN` set. 2. Trigger a controlled non-secret test error or panic in a development-only path or test harness. 3. Verify the event appears in Sentry with environment `development`. 4. Confirm no cookies, authorization headers, internal service secrets, request bodies, encrypted secret material, raw attestation documents, cloud credentials, SMTP credentials, or Paddle secrets appear in the event. 5. Clear the test hook before merging unless it is test-only and guarded. ### 3. Production environment Production should use real release identifiers and conservative sampling. Recommended production runtime `.env` additions: ```dotenv SENTRY_DSN=https://<backend-public-key>@<sentry-host>/<backend-project-id> SENTRY_ENVIRONMENT=production SENTRY_RELEASE=<git-sha-or-release-version> SENTRY_TRACES_SAMPLE_RATE=0 SENTRY_PROFILES_SAMPLE_RATE=0 SENTRY_SEND_DEFAULT_PII=false VITE_SENTRY_DSN=https://<frontend-public-key>@<sentry-host>/<frontend-project-id> VITE_SENTRY_ENVIRONMENT=production VITE_SENTRY_RELEASE=<git-sha-or-release-version> VITE_SENTRY_TRACES_SAMPLE_RATE=0 ``` Production rollout guidance: 1. Deploy backend Sentry first with `SENTRY_TRACES_SAMPLE_RATE=0` and frontend tracing disabled. 2. Verify events are grouped by service and release. 3. Verify sensitive data scrubbing against real-shaped but fake test requests. 4. Only after error capture is stable, consider enabling a very low traces sample rate such as `0.01`, and only for routes where privacy and volume are understood. 5. Keep Session Replay disabled unless a separate privacy/security review approves it. 6. Set Sentry alert routing for production only; keep development alerts muted or routed to a low-priority channel. ### 4. Source maps and releases Frontend source maps should be uploaded from CI/release automation, not from runtime containers. If source map upload is added later: - use `@sentry/vite-plugin` or `sentry-cli` in CI; - store `SENTRY_AUTH_TOKEN` only as a CI secret; - associate source maps with `VITE_SENTRY_RELEASE`; - delete or avoid publishing source maps in served static assets if that is the desired production policy; - do not commit, print, or bake `SENTRY_AUTH_TOKEN` into images. ## Security and privacy requirements Sentry must not send or store: - request bodies or response bodies; - cookies; - `Authorization` headers; - `INTERNAL_SERVICE_SECRET`; - Paddle API keys, webhook secrets, checkout payload secrets, or billing provider secrets; - SMTP credentials; - cloud credentials; - plaintext `.env` values; - encrypted secret material from `.caution/secrets/*.asc`; - Locksmith shard contents or private keys; - raw attestation documents, nonce material, PCR blobs, or certificate chains unless explicitly reviewed and redacted; - customer workload secrets or deployment-time secrets; - private SSH keys, WebAuthn challenge responses, or passkey credential material. Do not initialize Sentry inside enclave/customer workload code or deterministic/reproducible build paths. Sentry should be limited to operator-facing platform services and the dashboard. ## Acceptance criteria - `SENTRY_DSN` unset or empty means backend Sentry is fully disabled and services behave as they do today. - `VITE_SENTRY_DSN` unset or empty means frontend Sentry is fully disabled. - Backend services emit Sentry events for panics and error-level tracing events when enabled. - Events include service, environment, and release context. - Request context uses matched routes/sanitized IDs rather than raw attacker-controlled values where possible. - Sensitive headers and known secret-bearing fields are scrubbed before events are sent. - Request/response bodies are not captured by default. - Frontend captures uncaught Vue/browser errors when enabled, with `sendDefaultPii: false` and replay disabled. - `env.example`, deployment docs, or relevant README sections document dev and prod Sentry setup. - Tests cover configuration parsing/defaults and scrubbing behavior. - Existing regular checks continue to pass. ## Suggested test plan Backend: ```bash cargo check -p api -p gateway -p metering -p email-service cargo test -p caution-observability cargo test --workspace ``` Frontend: ```bash cd frontend npm install npm run build ``` Container/build smoke tests where feasible: ```bash make build-api make build-gateway make build-metering make build-email ```

caution-bot commented 2026年07月10日 22:20:43 +02:00

Duplicate/related issue audit while moving this into General Availability:

• Supersedes #132 (Error tracking and logging). That issue asked for durable error tracking/logging, and the later discussion specifically called out Sentry as the likely error-tracking solution. Closing #132 in favor of this more detailed GA-scoped issue.
• Includes the already-closed duplicate lineage from #247 (Add instrumentation for error notifications), which had previously been closed in favor of #132.

I also checked #233 (Observability Into Builders). That one is related but not a duplicate: it focuses on builder logs/metrics and encrypted log streaming, while this issue intentionally limits Sentry to operator-facing platform services/dashboard and says not to initialize Sentry inside builder/enclave/reproducible paths.

Duplicate/related issue audit while moving this into General Availability: - Supersedes #132 (`Error tracking and logging`). That issue asked for durable error tracking/logging, and the later discussion specifically called out Sentry as the likely error-tracking solution. Closing #132 in favor of this more detailed GA-scoped issue. - Includes the already-closed duplicate lineage from #247 (`Add instrumentation for error notifications`), which had previously been closed in favor of #132. I also checked #233 (`Observability Into Builders`). That one is related but not a duplicate: it focuses on builder logs/metrics and encrypted log streaming, while this issue intentionally limits Sentry to operator-facing platform services/dashboard and says not to initialize Sentry inside builder/enclave/reproducible paths.