-
Notifications
You must be signed in to change notification settings - Fork 1.4k
-
Hi Everyone, I am trying to run SB in parallel. in this code I am expecting output to be:
URL: https://google.com/ncr - Title: Google
URL: https://www.python.org - Title: Welcome to Python.org
URL: https://www.example.com - Title: Example Domain
URL: https://www.wikipedia.org - Title: Wikipedia
URL: https://www.github.com - Title: GitHub · Build and ship software on a single, collaborative platform · GitHub
but as soon as I activate uc=True or even with uc_subprocess=True. I even tried passing user_data_dir also, it gives out all sorts of problems like this:
URL: https://google.com/ncr - Title: Google
URL: https://www.python.org - Title: Wikipedia
URL: https://www.example.com - Title: Example Domain
URL: https://www.wikipedia.org - Title: None
URL: https://www.github.com - Title: GitHub · Build and ship software on a single, collaborative platform · GitHub
or this
URL: https://google.com/ncr - Title: GitHub · Build and ship software on a single, collaborative platform · GitHub
URL: https://www.python.org - Title: GitHub · Build and ship software on a single, collaborative platform · GitHub
URL: https://www.example.com - Title: Example Domain
URL: https://www.wikipedia.org - Title: GitHub · Build and ship software on a single, collaborative platform · GitHub
URL: https://www.github.com - Title: GitHub · Build and ship software on a single, collaborative platform · GitHub
Or even sometimes only one window is opening all urls, and others getting stalled. and does not even quit.
from seleniumbase import SB import concurrent.futures def run_instance(url): with SB(uc=True, uc_subprocess=True) as sb: sb.open(url) sb.sleep(5) title = sb.get_page_title() return f"URL: {url} - Title: {title}" if __name__ == '__main__': # List of URLs to process urls = [ "https://google.com/ncr", "https://www.python.org", "https://www.example.com", "https://www.wikipedia.org", "https://www.github.com", # Add more URLs as needed ] # Run the instances with 5 parallel workers with concurrent.futures.ProcessPoolExecutor(max_workers=5) as executor: results = executor.map(run_instance, urls) # Print the title retrieved from each URL for result in results: print(result)
so I suppose UC mode does not support multithreading or multiprocessing.
is there a way I can parallelize opening urls in different windows using UC mode?
Beta Was this translation helpful? Give feedback.
All reactions
See the example for multithreading with CDP Mode:
SeleniumBase/examples/cdp_mode/raw_multi_cdp.py
Also see SeleniumBase/examples/raw_multi_drivers.py
You'll need to use ThreadPoolExecutor.
Replies: 1 comment
-
See the example for multithreading with CDP Mode:
SeleniumBase/examples/cdp_mode/raw_multi_cdp.py
Also see SeleniumBase/examples/raw_multi_drivers.py
You'll need to use ThreadPoolExecutor.
Beta Was this translation helpful? Give feedback.