-
Notifications
You must be signed in to change notification settings - Fork 1.4k
-
Hello everyone,
Please help me to scrape this website https://ahrefs.com/traffic-checker/
I am using Seleniumbase version - seleniumbase==4.30.8
While running locally everything is fine, but on docker I face a lot of different error which include:
Screenshot 2024年10月04日 at 14 59 23
The problem is that usually, Cloudflare should start after I click the check traffic button. Other errors are:
seleniumbase.common.exceptions.NoSuchElementException: Message:
celery_1 | Element {input[placeholder="Enter domain or URL"]} was not present after 20 seconds!
Exception: Text {Checking} in {html} was still visible after 12 seconds!
seleniumbase.common.exceptions.NoSuchElementException: Message:
celery_1 | Element {span:contains("Check traffic")} was not present after 20 seconds!
The problem is that the page contains those elements, but Seleniumbase cannot find those elements.
My script is almost the same as on the Seleniumbase docs:
with SB(uc=True, test=True, incognito=True, xvfb=True, locale_code="en") as sb:
url = "https://ahrefs.com/traffic-checker"
input_field = 'input[placeholder="Enter domain or URL"]'
submit_button = 'span:contains("Check traffic")'
sb.uc_open_with_reconnect(url) # The bot-check is later
sb.uc_gui_click_captcha()
sb.wait_for_element_visible(input_field, timeout=30)
sb.type(input_field, domain)
sb.reconnect(0.5)
sb.wait_for_element_visible(submit_button, timeout=30)
sb.uc_click(submit_button, reconnect_time=4)
sb.uc_gui_click_captcha()
sb.wait_for_text_not_visible("Checking", timeout=30)
page_source = sb.get_page_source()
Another error I am getting is packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 199, in write_and_convert text = self.convert_osc(text) RecursionError: maximum recursion depth exceeded at 2024年10月04日 10:44:03.746886+00:00
Thank you in advance for your answers!
Beta Was this translation helpful? Give feedback.
All reactions
Answered by
mdmintz
Oct 4, 2024
This script is working for me:
from seleniumbase import SB with SB(uc=True, test=True, incognito=True, locale_code="en") as sb: url = "https://ahrefs.com/website-authority-checker" input_field = 'input[placeholder="Enter domain"]' submit_button = 'span:contains("Check Authority")' sb.uc_open_with_reconnect(url) # The bot-check is later sb.type(input_field, "github.com/seleniumbase/SeleniumBase") sb.reconnect(0.1) sb.uc_click(submit_button, reconnect_time=3.25) sb.uc_gui_click_captcha() sb.wait_for_text_not_visible("Checking", timeout=11.5) sb.highlight('p:contains("github.com/seleniumbase/SeleniumBase")') sb.highlight('a:contains("Top 100 back...
Replies: 1 comment 5 replies
-
This script is working for me:
from seleniumbase import SB with SB(uc=True, test=True, incognito=True, locale_code="en") as sb: url = "https://ahrefs.com/website-authority-checker" input_field = 'input[placeholder="Enter domain"]' submit_button = 'span:contains("Check Authority")' sb.uc_open_with_reconnect(url) # The bot-check is later sb.type(input_field, "github.com/seleniumbase/SeleniumBase") sb.reconnect(0.1) sb.uc_click(submit_button, reconnect_time=3.25) sb.uc_gui_click_captcha() sb.wait_for_text_not_visible("Checking", timeout=11.5) sb.highlight('p:contains("github.com/seleniumbase/SeleniumBase")') sb.highlight('a:contains("Top 100 backlinks")') sb.set_messenger_theme(location="bottom_center") sb.post_message("SeleniumBase wasn't detected!")
Note that UC Mode is not fully compatible with Docker because Docker adds a fingerprint that makes UC Mode detectable.
Beta Was this translation helpful? Give feedback.
All reactions
5 replies
-
Thanks for the reply!!! I have also noticed this:
- Element not found error appears even if the element exists in page
- I have a related error. That is what it looks like. self.__convertor.write(text)
celery_1 | File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write
celery_1 | self.write_and_convert(text)
celery_1 | File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert
celery_1 | self.write_plain_text(text, cursor, len(text))
celery_1 | File "...", line -1, in [rest of traceback truncated]
celery_1 | RecursionError: maximum recursion depth exceeded - Also, I have noticed that errors opened with sb do not raise errors in docker, can you explain me why it might happen?
Beta Was this translation helpful? Give feedback.
All reactions
-
- If the element is not found, then your selector may be incorrect. Note that iframes must be switched into first to see in there. And note that closed Shadow DOM is not reachable by Selenium.
celeryis an external program. This space is for SeleniumBase questions only.- Docker environments are different from regular environments. Save those questions for the Docker people.
Beta Was this translation helpful? Give feedback.
All reactions
-
That is the script that I have used:
try:
with SB(uc=True, test=True, incognito=True, xvfb=True, locale_code="en") as sb:
url = "https://ahrefs.com/traffic-checker"
input_field = 'input[placeholder="Enter domain or URL"]'
submit_button = 'span:contains("Check traffic")'
sb.uc_open_with_reconnect(url) # The bot-check is later
sb.uc_gui_click_captcha()
# sb.switch_to_default_content() # Return to default frame after solving
sb.wait_for_element_visible(input_field, timeout=20)
sb.type(input_field, domain)
sb.reconnect(0.1)
sb.wait_for_element_visible(submit_button, timeout=20)
sb.uc_click(submit_button, reconnect_time=4)
sb.uc_gui_click_captcha()
sb.wait_for_text_not_visible("Checking", timeout=12)
page_source = sb.get_page_source()
soup = BeautifulSoup(page_source, 'html.parser')
print('everything is finished')
except Exception as e:
print('exception handled')
- From this sometimes I have got this error:
Element {span:contains("Check traffic")} was not present after 20 seconds!
By 'element' I meant such cases when element is present, but sometimes seleniumbase was not able to detect it. - Second question was related colorama library which is used in seleniumbase as I have understand. Is it possible to completely ignore or remove it from seleniumbase implementation?
File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write - Let me give you an example what I meant by this:
This code cannot detect 'chec detect'
try:
with SB(uc=True, test=True, incognito=True, xvfb=True, locale_code="en") as sb:
url = "https://ahrefs.com/traffic-checker"
input_field = 'input[placeholder="Enter domain or URL"]'
submit_button = 'span:contains("Chec traffic")'
sb.uc_open_with_reconnect(url) # The bot-check is later
sb.uc_gui_click_captcha()
# sb.switch_to_default_content() # Return to default frame after solving
sb.wait_for_element_visible(input_field, timeout=20)
sb.type(input_field, domain)
sb.reconnect(0.1)
sb.wait_for_element_visible(submit_button, timeout=20)
sb.uc_click(submit_button, reconnect_time=4)
sb.uc_gui_click_captcha()
sb.wait_for_text_not_visible("Checking", timeout=12)
page_source = sb.get_page_source()
soup = BeautifulSoup(page_source, 'html.parser')
print('everything is finished')
except Exception as e:
print('exception handled')
Logically in Selenium in that case Exception is raised, but in Seleniumbase this situation is different:
Screenshot 2024年10月04日 at 21 37 58
I run this code on python file locally just to show what I meant. All those issues are directly connected Selniumbase, that is why your opinion and answers really matters for me!
Thank you for your responses, and for this package!!!!
Beta Was this translation helpful? Give feedback.
All reactions
-
I will be really grateful if can help me with the questions above!!
Beta Was this translation helpful? Give feedback.
All reactions
-
Note that UC Mode is not fully compatible with Docker because Docker adds a fingerprint that makes UC Mode detectable
Does this mean we won’t be able to use SeleniumBase with Docker?
Beta Was this translation helpful? Give feedback.
All reactions
Answer selected by
mdmintz
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment