Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

♻️ Refactor Cloudinary upload logic to add a upload retry #88

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
colbyfayock merged 6 commits into cloudinary-community:main from taciturnaxolotl:upload-retry
Nov 10, 2023

Conversation

@taciturnaxolotl
Copy link
Contributor

@taciturnaxolotl taciturnaxolotl commented Oct 14, 2023
edited
Loading

Hopefully this works :)

Fixes #82

Copy link

netlify bot commented Oct 14, 2023
edited
Loading

Deploy Preview for netlify-plugin-cloudinary ready!

Name Link
🔨 Latest commit 894b2dc
🔍 Latest deploy log https://app.netlify.com/sites/netlify-plugin-cloudinary/deploys/654e893a26f4ca00082c55f8
😎 Deploy Preview https://deploy-preview-88--netlify-plugin-cloudinary.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

Copy link
Collaborator

hey @kcoderhtml clever idea of using a loop like that, hadnt considered that

i think we're running into an issue though, where the retry logic doesn't seem to be running for me

to test, i replaced fullPath in both upload lines to something random and invalid like /

i think what's happening is that when the first error occurs, the catch re-throws the error, resulting in getCloudinaryUrl throwing the error and never hitting the second attempt, though i certainly may be wrong

i wasn't able to get console.log('attempt', attempt) logged out more than once by placing it at the of the loop

i'm wondering if the loop needs to be inside of the try catch, or perhaps inside of the catch, you first check the attempt number and seeing if it's at the maximum number of attempts before throwing (which we could possibly set up for configuration later)

any chance you were able to get this running locally? if not and you're willing to try, happy to help you get set up if the instructions on the README didnt work, that way you can test it out

Copy link
Contributor Author

I would be happy to help but could not get the demo site to run, I kept getting this error: https://app.warp.dev/block/AK6xV3BiPb68w5XM3cyhh7.

Copy link
Collaborator

oh weird - is that after building the project? since it needs to compile the TS, otherwise the files wont exist in dist which is the output folder

Copy link
Contributor Author

I just tried building, but I got this error: https://app.warp.dev/block/ZpMZMoKUYbvYa2Zv1Qy24Y. I also tried with npm run build but it wanted me to install the node modules, but they wouldn't install: https://app.warp.dev/block/UV6d9X6VzdjyV49kvpYiSi. I'm sure I'm missing something, so thank you for your help.

Copy link
Collaborator

how about try the following

in the root:

pnpm install
pnpm build

then i usually test with this command:

netlify deploy --build --filter docs --context production

i run it directly instead of trying to use the package.json script

production might not be required but it helps point to images that are on the production site, and sometimes building locally can be problematic

im not sure if you set it up yet but it requires the Netlify CLI and logged in to an account

if you dont have any luck, i wonder if it's reproducible by adding a test for getCloudinaryUrl with it

Copy link
Contributor Author

The pnpm build fails with You must supply a cloudName when initializing the asset.

Copy link
Collaborator

colbyfayock commented Oct 19, 2023
edited
Loading

yeah, you need to configure the cloud name in order for it to work as it ultimately uploads and uses Cloudinary, which requires an account

to upload though you'll need to add the api key and secret as well

would you rather just create a test for this?

otherwise, once you connect your netlify site through the CLI, you can add your environment variables in the Netlify dashboard, those should be dynamically pulled in when trying to deploy locally

Copy link
Collaborator

@kcoderhtml were you going to continue on this? if this is wrapped up by mid-November we can still consider this Hackoberfest given it was opened within October

Copy link
Contributor Author

Sorry, I've been quite busy these last few weeks. I am still working on this and will be writing a test for the code piece.

Copy link
Collaborator

awesome sounds great! thank you, let me konw if you need a hand

Copy link
Contributor Author

thanks! will do.

Copy link
Collaborator

hey @kcoderhtml when you get back to this, please merge in main as it has some updates in the main index.ts related to concurrency. hoping it doesnt conflict with your work

PR: #89

The code now properly handles upload errors by retrying the upload up to 3 times. If an error occurs during an upload attempt, the error message is logged and the code waits for 500ms before retrying.
Copy link
Contributor Author

I think that might have done it! There is a new timeout and maxRetries settings to adjust to your liking.

Copy link
Collaborator

awesome! the retries worked great

im seeing one issue though that makes me think there's a loose async request somewhere, but i can't tell if its your code or the wraping code

image

if you notice, the 3rd attempts are happening in a later build stage which i think could potentially cause issues if there's a lot of uploads

im goign to play around with it a little more likely next week to try to determine the cause of that, but this code genrally is looking great

Copy link
Collaborator

yeah its definitely related to this code somehow. i put a before/after console log around the async function and without the code, it all wraps up before continuing

image

but with the code, it never ends

image

everything looks sounds to me though, i wonder if something is happening

the way im testing this btw is replacing fullPath with garbage ./asdf and it fails automatically.

Copy link
Collaborator

i think i figured it out and i dont think its actually related to this - ithink the error that's being thrown from within getCloudinaryUrl is not being properly caught anymore from within the wrapping code and thats causing the issue. im gonna separately get a bug fix in

taciturnaxolotl reacted with thumbs up emoji

Copy link
Collaborator

apologies for the confusion there but we're good to go!

thanks @kcoderhtml great add here

@colbyfayock colbyfayock merged commit bda0cd9 into cloudinary-community:main Nov 10, 2023
@taciturnaxolotl taciturnaxolotl deleted the upload-retry branch November 10, 2023 19:55
Copy link
Collaborator

@all-contributors please add @kcoderhtml for code

Copy link
Contributor

@colbyfayock

I've put up a pull request to add @kcoderhtml! 🎉

Copy link

🎉 This PR is included in version 1.14.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Copy link
Contributor Author

Thank you! Will this be included in cloudinary hacktoberfest contributions? The swag pack looks pretty sweet!

colbyfayock reacted with hooray emoji

Copy link
Collaborator

@kcoderhtml congrats on the merged PR! since you opened this in October, you've qualified for Hacktoberfest swag! please email hacktoberfest@cloudinary.com with your github username and a link to your contribution where I'll follow up to request more information for getting you your swag

https://cloudinary.com/blog/hacktoberfest-celebrate-open-source-sdks

colbyfayock pushed a commit that referenced this pull request Nov 10, 2023
Adds @kcoderhtml as a contributor for code.
This was requested by colbyfayock [in this
comment](#88 (comment))
[skip ci]
---------
Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com>
Copy link
Contributor Author

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Feature] Add retry logic for failed uploads

AltStyle によって変換されたページ (->オリジナル) /