-
Notifications
You must be signed in to change notification settings - Fork 5.1k
Detect and log Docker Hub rate limit errors in image pulls #21590
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Detect and log Docker Hub rate limit errors in image pulls #21590
Conversation
[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: henry3260
Once this PR has been reviewed and has the lgtm label, please assign prezha for approval. For more information see the Code Review Process.
The full list of commands accepted by this bot can be found here.
Approvers can indicate their approval by writing /approve
in a comment
Approvers can cancel approval by writing /approve cancel
in a comment
Hi @henry3260. Thanks for your PR.
I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test
on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.
Once the patch is verified, the new status will be reflected by the ok-to-test
label.
I understand the commands that are listed here.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
Can one of the admins verify this patch?
pkg/minikube/machine/cache_images.go
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we should use
out.Styled(style.Notice, Kicbase images have not been deleted. To delete images run:
)
I supsect there are other places we could that
do you putting a Before/Aeter this PR,
you can simulate the rate limit by making a Dummy docker binary that always returns with rate limit...
howevever we also have pulling images using go library , you mock those libraries for full coverage of everywhere we pull image
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
also in our addons we need to check if addons which docker images if they are rate limitted or not.
so we should add code to where we enable the addons (and verify that the image is not rate limitted) and if it is we should tell the users nicely
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, thanks for the clarification!
For this PR, I can switch the logging in cache_images.go
from klog
to out.Styled(style.Notice, ...)
where appropriate.
Do you suggest I update all klog
calls in that file, or just the ones related to rate limit messages?
Also, I’ll look into adding tests by mocking docker/go-containerregistry to simulate the rate limit scenario.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Got it!
So for addons, I’ll add a check when enabling them to verify the image is not rate-limited, and if it is, we’ll surface a friendly notice to the user.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we should use out.Styled(style.Notice,
Kicbase images have not been deleted. To delete images run:
)I supsect there are other places we could that
do you putting a Before/Aeter this PR,
you can simulate the rate limit by making a Dummy docker binary that always returns with rate limit... howevever we also have pulling images using go library , you mock those libraries for full coverage of everywhere we pull image
Thanks for the suggestion! I'm not sure how to properly mock go-containerregistry
's remote.Image
in our current test setup. Could you give me a pointer or a small example of how you'd recommend doing that?
So far, I tried:
- Using a dummy Docker binary (which worked for simulating errors).
- Replacing the function with a variable to trigger the error path, but it didn't enter the
if
block as expected.
At this point, I'm not sure how to apply go-containerregistry
to simulate the rate-limit case in tests. Any guidance or example would be really helpful.
If you'd prefer, I can open a follow-up issue for this discussion.
pkg/minikube/machine/cache_images.go
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Using strings.Contains() means we are not using a public API. This is too fragile to depend on. Do we have an error code field or error name we can check instead?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Using
strings.Contains()
means we are not using a public API.
This is too fragile to depend on.
Do we have an error code field or error name we can check instead?
Thanks for the feedback! 🙏
I agree that relying on strings.Contains()
is fragile.
In my opinion, we could consider using go-containerregistry, since its API exposes structured errors. For example:
if terr, ok := err.(*transport.Error); ok { if terr.StatusCode == http.StatusTooManyRequests { // This confirms it's a 429 rate limit error } }
This way we can explicitly check the HTTP status code instead of depending on error strings.
Do you think this approach would be acceptable for minikube?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks good, this is way to handle errors.
6b32174
to
5973660
Compare
fb7f9be
to
f3a7a94
Compare
Uh oh!
There was an error while loading. Please reload this page.
This PR improves error visibility when pulling images fails due to Docker Hub rate limits.
Changes
PullImages
incache_images.go
to detect429 Too Many Requests
errors and return them to the caller.Impact
PullImages
can now handle the429
error explicitly.Testing
cache_image_test.go
that simulates a429 Too Many Requests
error inpullImages
.Docker Hub rate limit reached for profile minikube. See: https://www.docker.com/increase-rate-limit
Before:
imageAfter:
imageFixes: #21442