Seleniumbase gets detected by imperva · seleniumbase/SeleniumBase · Discussion #3106

SpielerNogard
Sep 8, 2024

Hey since today imperva seems to detect seleniumbase in uc mode.
the site im crawling is https://join.pokemon.com
and since today a new script gets injected, which removes half os the site. But only when using Brave or when the browser is started using a script (both webbrowser or seleniumbase)

Does someone have any ideas ?
Blocking this script does not work

Replies: 4 comments 7 replies

mdmintz
Sep 8, 2024
Maintainer

Follow the example from SeleniumBase/examples/raw_antibot_login.py, which demonstrates how to avoid detection on sites using real-time bot-scanning software (such as Brotector, Imperva, and DataDome):

from seleniumbase import SB
with SB(uc=True, test=True) as sb:
 url = "https://seleniumbase.io/antibot/login"
 sb.uc_open_with_disconnect(url, 2.15)
 sb.uc_gui_write("\t" + "demo_user")
 sb.uc_gui_write("\t" + "secret_pass")
 sb.uc_gui_press_keys("\t" + " ") # For Single-char keys
 sb.sleep(1.5)
 sb.uc_gui_press_keys(["\t", "ENTER"]) # Multi-char keys
 sb.reconnect(1.8)
 sb.assert_text("Welcome!", "h1")
 sb.set_messenger_theme(location="bottom_center")
 sb.post_message("SeleniumBase wasn't detected!")

There, sb uses special PyAutoGUI calls to avoid detection while the driver is disconnected from the browser. Once it's safe to reconnect, do so (as shown), and then you can resume using Selenium calls. This is a bit of a hack right now, as you need to know how many times to press the Tab key before typing text or clicking a button.

Here's another example, SeleniumBase/examples/raw_brotector_captcha.py:

from seleniumbase import SB
with SB(uc=True, test=True) as sb:
 url = "https://seleniumbase.io/apps/brotector"
 sb.uc_open_with_disconnect(url, 2.2)
 sb.uc_gui_press_key("\t")
 sb.uc_gui_press_key(" ")
 sb.reconnect(2.2)

Here's a slightly longer, more fun example: SeleniumBase/examples/raw_hobbit.py:

from seleniumbase import SB
with SB(uc=True, test=True) as sb:
 url = "https://seleniumbase.io/hobbit/login"
 sb.uc_open_with_disconnect(url, 2.2)
 sb.uc_gui_press_keys("\t ")
 sb.reconnect(1.5)
 sb.assert_text("Welcome to Middle Earth!", "h1")
 sb.set_messenger_theme(location="bottom_center")
 sb.post_message("SeleniumBase wasn't detected!")
 sb.click("img")
 sb.sleep(5.888) # Cool animation happening now!

I'm currently working on a way to use selectors while disconnected so that tabbing through elements isn't needed anymore to avoid detection.

0 replies

SpielerNogard
Sep 8, 2024
Author

Thanks, didnt know uc_open_with_disconnect exists. Sadly this also does not work und gets detected.

This is how the site should look, when nothing gets detected
Screenshot 2024年09月08日 at 17 27 55

And here is, when it gets detected
Screenshot 2024年09月08日 at 17 30 03

Minimal code example:

from seleniumbase import Driver
import time
kwargs = {}
chromium_args = "--lang=en-us"
kwargs["chromium_arg"] = chromium_args
driver = Driver(uc=True, headless2=False, **kwargs)
driver.uc_open_with_disconnect("https://join.pokemon.com/", 2.2)
time.sleep(200)

Im open for any other ideas

2 replies

@mdmintz

mdmintz Sep 8, 2024
Maintainer

This worked for me:

from seleniumbase import SB
with SB(uc=True, incognito=True) as sb:
 url = "https://join.pokemon.com/"
 sb.uc_open_with_disconnect(url)
 breakpoint()

Screenshot 2024年09月08日 at 11 39 07 AM

@SpielerNogard

SpielerNogard Sep 8, 2024
Author

Interresting, for me not

SpielerNogard
Sep 8, 2024
Author

Ok i have

Google Chrome:
Version 128.0.6613.120 (Official Build) (arm64)

Python:
Python 3.11.5

selenium 4.24.0
seleniumbase 4.30.3

And MacOS 14.6.1 (23G93)

Do you have any more ideas ?

1 reply

@mdmintz

mdmintz Sep 8, 2024
Maintainer

You can try https://github.com/ultrafunkamsterdam/nodriver

pip install nodriver

import nodriver
async def main():
 browser = await nodriver.start()
 page = await browser.get("https://join.pokemon.com/")
 breakpoint()
if __name__ == "__main__":
 nodriver.loop().run_until_complete(main())