Linked Questions

18 questions linked to/from Python selenium multiprocessing
0 votes
3 answers
955 views

I'm trying to run two(or more) selenium webdrivers with Python at the same time I have so far tried using Python's Multiprocessing module, I have used it this way: def _(): sets = list() pool =...
0 votes
0 answers
200 views

I'm wanting to write a script in python, with selenium, to automate logging into a website and performing certain tasks on multiple accounts in parallel. I'm taking the logins from a text file with ...
71 votes
4 answers
86k views

I'm using Scrapy to crawl a webpage. Some of the information I need only pops up when you click on a certain button (of course also appears in the HTML code after clicking). I found out that Scrapy ...
1 vote
2 answers
16k views

I am using selenium and chrome-driver to scrape data from some pages and then run some additional tasks with that information (for example, type some comments on some pages) My program has a button. ...
3 votes
3 answers
5k views

I am using selenium and Python to do a big project. I have to go through 320.000 webpages (320K) one by one and scrape details and then sleep for a second and move on. Like bellow: links = ["...
6 votes
2 answers
4k views

I have a use case for which I'm unable to develop a logic. Floating it here for recommendations from experts. Quick context: I have a list of 2,500 URLs. I am able to scrape them sequentially using ...
1 vote
1 answer
2k views

I have around 30k license numbers that I want to search from a website and extract all the relevant information from it When I tried the extracting the information from the function below by looping ...
3 votes
1 answer
1k views

I'm trying to scrape a javascript website using scrapy and selenium. I open the javascript website using selenium and a chrome driver and I scrape all the links to different listings from the current ...
0 votes
2 answers
1k views

I have a script that creates multiple selenium.webdriver-instances, executes a js-script and reads the resulting logs of them. Most of the time the script runs without problems, but in a few cases the ...
0 votes
2 answers
946 views

I currently have a selenium function which does the following summary of the code: def (list): FOR LOOP in list: # Page A (initial), Contains 12 requests,bs4 grabs element coordinates. [...
-1 votes
2 answers
605 views

I've written a script using selenium implementing multiprocessing within it taking the idea of this answer. The script works just fine and I see all the results in the console. However, when the ...
0 votes
1 answer
494 views

i have a task where i need selenium automation to do multiple searches inputs, each has to open the browser do some interactions and close, i can do that one after the other , but i thought that if i ...
-2 votes
3 answers
768 views

How can I execute the below script using multiple browsers? Every n urls should be executed using a separate browser. I should be able to define the value of n (parallel scraping) import pandas as pd ...
user avatar
0 votes
2 answers
524 views

I want to use ThreadPoolExecutor to parallelize some (legacy) code with database access. I would like to avoid creating a new database connection for each thread, so I use threading.local() to keep a ...
1 vote
2 answers
238 views

I am trying to create a method that will create threads and send them into a thread pool. How do I stop individual threads after creating them? edit: This is being used for webscraping and will need ...

15 30 50 per page
1
2