9
2
Fork
You've already forked platform
2

[EXPERIMENTAL] builder: add opt-in S3 BuildKit layer cache for EIF builds #353

Open
vkobel wants to merge 3 commits from fix/eif-s3-layer-cache into main
pull from: fix/eif-s3-layer-cache
merge into: caution:main
caution:main
caution:feat/paddle-byoc-subscriptions
caution:feat/legal-publish-doc
caution:fix/issue-116-duplicate-ssh-key
caution:fix/issue-307-remove-apps
caution:fix/issue-265-http-port-private
caution:fix/issue-330-remove-build-binary
caution:feat/issue-164-account-id
caution:fix/issue-119-logout-confirmation
caution:feat/add-tofu-lockfile-cache
caution:feat/webauthn-discoverable-login
caution:docs/issue-339-install-instructions
caution:fix/issue-336-split-qr-login-tokens
caution:fix/verify-app-source-preflight
caution:fix/many-git-push-deploy
caution:fix/issue-334-email-verification-throttle
caution:fix/issue-273-billing-url
caution:fix/issue-333-final-usage-collection
caution:fix/issue-270-low-credit-email
caution:fix/issue-301-remove-auto-topup
caution:fix/issue-320-byoc-auth-first
caution:fix/issue-348-git-progress-checklist
caution:fix/issue-321-byoc-caution-hcl
caution:fix/issue-332-resource-id-suspension
caution:fix/eif-app-mode-normalization
caution:feat/cli-stagex-container
caution:fix/authz-role-checks
caution:fix/paddle-customdata-credit-mint
caution:fix/cli-hangs-unsupported-ssh-url
caution:feat/print-platform-commit-dashboard
caution:anton/feat/multi-user
caution:fix/config-domain-validation
caution:anton/support-ssh-destroy-get
caution:fix/cli-config-dir
caution:feat/publish-build-inputs
caution:feat/locksmith-polishing
caution:fix/procfile-run-command-no-split
caution:anton/byoc-require-login
caution:fix/hcl-init-template
caution:feat/egress-default-deny
caution:ryansquared/hcl
caution:quotas
caution:anton/feat/region-routing-resource-unavailable-alert
caution:fix/attestation-502-dashboard
caution:ryansquared/systemdize
caution:fix/verify-git-credential-hang
caution:anton/fix-low-credits-alert
caution:xenushka/rename-byoc
caution:ryansquared/add-download-eif
caution:add-debug-flag
caution:test/pr1-integration-harness
caution:anton/fix/only-pass-required-env-vars
caution:ryansquared/add-database-bucket

LLM Experiment to see if this is an actual way to speed things up a bit

Should test in dev.


Adds an opt-in S3 BuildKit layer cache on remote builders, cutting the "Docker image built → EIF built" phase (~2m of a plain-app deploy) by reusing cached stages across builds — including the steve/locksmith/bootproof cargo builds when their pinned commits are unchanged.

  • Scoped & safe: cache runs only on the remote builder via a dedicated eifcache docker-container builder targeted with --builder (not --use), so the customer's docker build stays on the default driver — IMDS-isolated and loaded into the image store.
  • Zero impact on caution verify / local builds: they never set the cache env vars, so they take the byte-identical plain docker build path. Cache changes build speed, never output bytes → PCRs unaffected.
  • Bypass switch: build { layer_cache = false } in HCL forces a fully self-contained uncached build (escape hatch for a poisoned cache).

Requires: deploy updated api (new user-data) + apply the infra-bootstrap IAM change (buildcache/* S3 perms). Without the IAM grant, cache writes fail silently (ignore-error=true) and builds still succeed uncached.

Relates to #250 (Prebuild EnclaveOS components) — lighter-weight approach, no ECR image dependency.

Verified by compile + code review only; needs a real deploy trace to confirm the phase-time drop.

LLM Experiment to see if this is an actual way to speed things up a bit Should test in dev. --- Adds an opt-in S3 BuildKit layer cache on remote builders, cutting the "Docker image built → EIF built" phase (~2m of a plain-app deploy) by reusing cached stages across builds — including the steve/locksmith/bootproof cargo builds when their pinned commits are unchanged. - Scoped & safe: cache runs only on the remote builder via a dedicated eifcache docker-container builder targeted with --builder (not --use), so the customer's docker build stays on the default driver — IMDS-isolated and loaded into the image store. - Zero impact on caution verify / local builds: they never set the cache env vars, so they take the byte-identical plain docker build path. Cache changes build speed, never output bytes → PCRs unaffected. - Bypass switch: build { layer_cache = false } in HCL forces a fully self-contained uncached build (escape hatch for a poisoned cache). Requires: deploy updated api (new user-data) + apply the infra-bootstrap IAM change (buildcache/* S3 perms). Without the IAM grant, cache writes fail silently (ignore-error=true) and builds still succeed uncached. Relates to #250 (Prebuild EnclaveOS components) — lighter-weight approach, no ECR image dependency. Verified by compile + code review only; needs a real deploy trace to confirm the phase-time drop.
vkobel changed title from (削除) builder: add opt-in S3 BuildKit layer cache for EIF builds (削除ここまで) to [EXPERIMENTAL] builder: add opt-in S3 BuildKit layer cache for EIF builds 2026年07月02日 13:13:34 +02:00
Author
Owner
Copy link

Should we keep this as a temporary bandaid?

Should we keep this as a temporary bandaid?
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin fix/eif-s3-layer-cache:fix/eif-s3-layer-cache
git switch fix/eif-s3-layer-cache

Merge

Merge the changes and update on Forgejo.
git switch main
git merge --no-ff fix/eif-s3-layer-cache
git switch fix/eif-s3-layer-cache
git rebase main
git switch main
git merge --ff-only fix/eif-s3-layer-cache
git switch fix/eif-s3-layer-cache
git rebase main
git switch main
git merge --no-ff fix/eif-s3-layer-cache
git switch main
git merge --squash fix/eif-s3-layer-cache
git switch main
git merge --ff-only fix/eif-s3-layer-cache
git switch main
git merge fix/eif-s3-layer-cache
git push origin main
Sign in to join this conversation.
No reviewers
Labels
Clear labels
Compat/Breaking
Breaking change that won't be backward compatible
Component/Enclave-Builder
This issue affects Enclave Builder (git-push and CLI)
Component/Infrastructure
This issue affects generic Caution hosted infra components
Deploy/BYOC
Relates to Caution Bring-your-own-compute
Deploy/Full-managed
Relates to Caution fully managed cloud
Deploy/Self-hosted
Relates to Caution deployed on self-hosted infra
Interface/API
This issue affects the API
Interface/CLI
This issue affects the command line interface
Interface/Git
This issue affects the Git `push` interface
Interface/Web
This issue affects the web dashboard
Kind/Bug
Something is not working
Kind/Documentation
Documentation changes
Kind/Enhancement
Improve existing functionality
Kind/Feature
New functionality
Kind/Security
This is security issue
Kind/Testing
Issue or pull request related to testing
Priority
Critical
The priority is critical
Priority
High
The priority is high
Priority
Low
The priority is low
Priority
Medium
The priority is medium
Reviewed
Confirmed
Issue has been confirmed
Reviewed
Duplicate
This issue or pull request already exists
Reviewed
Invalid
Invalid issue
Reviewed
Won't Fix
This issue won't be fixed
Status
Abandoned
Somebody has started to work on this but abandoned work
Status
Blocked
Something is blocking this issue or pull request
Status
Need More Info
Feedback is required to reproduce issue or to continue work
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
caution/platform!353
Reference in a new issue
caution/platform
No description provided.
Delete branch "fix/eif-s3-layer-cache"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?