9
2
Fork
You've already forked platform
2

Fresh deployments fail health check, possible after internal attestation port move to 49502 #254

Closed
opened 2026年05月23日 22:26:02 +02:00 by vkobel · 2 comments

Fresh app deployments can fail after build with:

remote: [x] Waiting for health check...
remote: error: Enclave failed to become healthy: Attestation endpoint did not become healthy within 120 seconds

This started happening on apps that previously deployed successfully. Local QEMU debugging can still look fine, but remote deployment times out waiting for the attestation endpoint.

Possible cause: commit 318c08f0c9 moved internal platform ports from 8080/8081/8082 to 49500/49501/49502.

Hypothesis: the dedicated builder path and the deployer/user-data path may be on different revisions. If an EIF is built with bootproofd listening on 8082 but the host Caddy/vsock config forwards /attestation to 49502, or vice versa, /attestation never becomes healthy.

This would also explain why only some apps/deployments fail:

  • cached EIFs may have old internal ports (works with vanilla hello world demo, but not any other commit on top)
  • fresh EIFs may have new internal ports
  • builder instances and API/deployer images may not be deployed from the same commit;
  • the error message is generic and does not reveal whether Caddy health or /attestation failed.
Fresh app deployments can fail after build with: ```bash remote: [x] Waiting for health check... remote: error: Enclave failed to become healthy: Attestation endpoint did not become healthy within 120 seconds ``` This started happening on apps that previously deployed successfully. Local QEMU debugging can still look fine, but remote deployment times out waiting for the attestation endpoint. Possible cause: commit 318c08f0c9d09eb700b802684192ecd2c752b7e8 moved internal platform ports from 8080/8081/8082 to 49500/49501/49502. Hypothesis: the dedicated builder path and the deployer/user-data path may be on different revisions. If an EIF is built with bootproofd listening on 8082 but the host Caddy/vsock config forwards /attestation to 49502, or vice versa, /attestation never becomes healthy. This would also explain why only some apps/deployments fail: - cached EIFs may have old internal ports (works with vanilla hello world demo, but not any other commit on top) - fresh EIFs may have new internal ports - builder instances and API/deployer images may not be deployed from the same commit; - the error message is generic and does not reveal whether Caddy health or /attestation failed.
Owner
Copy link

Thanks for the detailed report @vkobel. Your analysis is correct: the failure can happen if the EIF and host-side user-data/Caddy/vsock config disagree on the reserved internal ports.

I bumped the component pins so the runtime services are aligned with the current reserved port range:

If the failure is caused by a stale cached EIF, adding no_cache: true to the Procfile will force a fresh EIF and should resolve it.

I'd suggest installing the latest CLI as well. Note that we are currently resolving some issues with locksmith as StageX is musl based and a lot of common Linux distros are glibc based so a subset of commands is currently broken, but will be fixed over the next 2 days.

Please let me know if your issue is resolved.

Thanks for the detailed report @vkobel. Your analysis is correct: the failure can happen if the EIF and host-side user-data/Caddy/vsock config disagree on the reserved internal ports. I bumped the component pins so the runtime services are aligned with the current reserved port range: - bootproof: https://git.distrust.co/public/bootproof/commit/78f531a2c245404a9d8879fb71cc397096ae0077 - locksmith: https://codeberg.org/caution/locksmith/commit/3e40803561c07bb31bc0a365bb53ca203f231f26 - steve: https://git.distrust.co/public/steve/commit/1dee1a0d55d1106fd8a237c09733406489cc8d12 If the failure is caused by a stale cached EIF, adding `no_cache: true` to the Procfile will force a fresh EIF and should resolve it. I'd suggest installing the latest CLI as well. Note that we are currently resolving some issues with locksmith as StageX is musl based and a lot of common Linux distros are glibc based so a subset of commands is currently broken, but will be fixed over the next 2 days. Please let me know if your issue is resolved.
Author
Owner
Copy link

Nice, thanks for the quick fix. Seems like it's working fine now after a few tests.

Nice, thanks for the quick fix. Seems like it's working fine now after a few tests.
Sign in to join this conversation.
No Branch/Tag specified
main
fix/issue-191-attestation-endpoint-copy
anton/feat/multi-user
feat/legal-publish-doc
fix/issue-119-logout-confirmation
fix/paddle-customdata-credit-mint
fix/issue-116-duplicate-ssh-key
fix/issue-265-http-port-private
fix/issue-330-remove-build-binary
feat/paddle-byoc-subscriptions
fix/issue-307-remove-apps
feat/issue-164-account-id
feat/add-tofu-lockfile-cache
feat/webauthn-discoverable-login
docs/issue-339-install-instructions
fix/issue-336-split-qr-login-tokens
fix/verify-app-source-preflight
fix/many-git-push-deploy
fix/issue-334-email-verification-throttle
fix/issue-273-billing-url
fix/issue-333-final-usage-collection
fix/issue-270-low-credit-email
fix/issue-301-remove-auto-topup
fix/issue-320-byoc-auth-first
fix/issue-348-git-progress-checklist
fix/issue-321-byoc-caution-hcl
fix/issue-332-resource-id-suspension
fix/eif-app-mode-normalization
feat/cli-stagex-container
fix/eif-s3-layer-cache
fix/authz-role-checks
fix/cli-hangs-unsupported-ssh-url
feat/print-platform-commit-dashboard
fix/config-domain-validation
anton/support-ssh-destroy-get
fix/cli-config-dir
feat/publish-build-inputs
feat/locksmith-polishing
fix/procfile-run-command-no-split
anton/byoc-require-login
fix/hcl-init-template
feat/egress-default-deny
ryansquared/hcl
quotas
anton/feat/region-routing-resource-unavailable-alert
fix/attestation-502-dashboard
ryansquared/systemdize
fix/verify-git-credential-hang
anton/fix-low-credits-alert
xenushka/rename-byoc
ryansquared/add-download-eif
add-debug-flag
test/pr1-integration-harness
anton/fix/only-pass-required-env-vars
ryansquared/add-database-bucket
No results found.
Labels
Clear labels
Compat/Breaking
Breaking change that won't be backward compatible
Component/Enclave-Builder
This issue affects Enclave Builder (git-push and CLI)
Component/Infrastructure
This issue affects generic Caution hosted infra components
Deploy/BYOC
Relates to Caution Bring-your-own-compute
Deploy/Full-managed
Relates to Caution fully managed cloud
Deploy/Self-hosted
Relates to Caution deployed on self-hosted infra
Interface/API
This issue affects the API
Interface/CLI
This issue affects the command line interface
Interface/Git
This issue affects the Git `push` interface
Interface/Web
This issue affects the web dashboard
Kind/Bug
Something is not working
Kind/Documentation
Documentation changes
Kind/Enhancement
Improve existing functionality
Kind/Feature
New functionality
Kind/Security
This is security issue
Kind/Testing
Issue or pull request related to testing
Priority
Critical
The priority is critical
Priority
High
The priority is high
Priority
Low
The priority is low
Priority
Medium
The priority is medium
Reviewed
Confirmed
Issue has been confirmed
Reviewed
Duplicate
This issue or pull request already exists
Reviewed
Invalid
Invalid issue
Reviewed
Won't Fix
This issue won't be fixed
Status
Abandoned
Somebody has started to work on this but abandoned work
Status
Blocked
Something is blocking this issue or pull request
Status
Need More Info
Feedback is required to reproduce issue or to continue work
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
caution/platform#254
Reference in a new issue
caution/platform
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?