holy crap @SuperKogito this went from 4-5 minutes to 44 seconds!! Major improvement!! https://github.com/USRSE/usrse.github.io/runs/5701139638?check_suite_focus=true

@SuperKogito I know you've been busy or not able to respond in the past - I've tested this (even via the action) on external repos so I'm going to merge for the super speed up, and we can open further issue/PR if any new issues arise! I hope you are doing well!

I am really sorry for taking some time, I intended to look at it today but couldn't. Overall the structure is nice (kudos for the new class), and the improvements are great ;) I only wonder if this might touch some requests limit if requests are simultaneous. However since the automatic tests didn't fail, I think it is safe to merge.

Oh yay you are around! I think we should actually be OK because it’s parsed on the level of the file (so checks are unique within a process run), and if we hit a case of shared urls across files it either will work off the bat still or retry. I tested here and on our USRSE repo and saw no issues so I think it’s a huge improvement worth it!

But I thought about this, and if we do need to handle duplication across jobs we could always parse urls first, taking this into account, and then run the multiprocess with no duplications.

That might further optimize actually - ok if O try this out and open another PR? And since I know you are around I will indeed wait for your review this time!

I agree, I think it is a great improvement 👏 I feel silly how we didn't think of it before 😝
yes filtering the duplicates before multiprocessing should make it even faster - Of course, give it a shot and if it is stable enough let's merge ;)

( just an idea )
I think if you have lengthy checks and looking for further faster processing, it might be worthy to revive #52 sine that will reach the limits of acceleration but asynchronous processing has its drawbacks and I think if we ever pursue that we will need to create different python and action folders (say: urltechie_super_speed :p ) cuz using that will force us to drop a lot of our useful generic features.

okay - I'm on it!

Agreed - let me get in a PR for optimizing the urls we check, and then let's rebase #52 and we can time the master branch against the same with async/aiohttp. If it's an improvement that is noticeable, it's definitely worth considering! If it's a trivial change then probably not worth the pain of async 😆