-
-
Notifications
You must be signed in to change notification settings - Fork 1.4k
CDP mode with shadowroot interaction #3552
-
Greetings,
I am happy to have discovered this well-maintained project and have been studying the docs now for a week and been conducting small tests.
During the past few days however, I've struggled with detecting / interacting with elements under one, or multiple open shadow root elements, which according to the example test files of seleniumbase repository should work, so I am forced to ask, if anyone could point out what am I doing wrong.
My issue as follows:
- In the below image you'll be able to see a semi-simple HTML structure with one shadow root at the top.
- I've tried to detect the button with:
self.selbase.cdp.find_elements(selector) // where selector = "onboarding-modal::shadow #w-button"`
- trying to click on it directly without finding it first:
self.selbase.cdp.gui_click_element(parent_selector)
try1: Element {onboarding-modal w-button[variant="secondary"]} was not found after 7 seconds!
try2: Element {#modal-div w-button[variant="secondary"]} was not found after 7 seconds!
My questions are:
- is it possible to detect and then click on the button element under open shadow root node?
- if yes, how? If not, how would you do it?
- is it possible to be done without disconnecting from CDP mode? If not and reconnection to the Webdriver is required, what does "safe way" to reconnect without detection mean in practice? The way I see it, it must mean that it has to be done while the connection to the website is closed?
(Note that reconnecting allows anti-bots to detect you, so only reconnect if it is safe to do so.)
Thank you in advance!
Beta Was this translation helpful? Give feedback.
All reactions
Replies: 3 comments 5 replies
-
As in the example below, you have to use sb.cdp.get_nested_element(parent_selector, selector) in order to get an element that's nested inside Shadow root elements.
from seleniumbase import SB with SB(uc=True) as sb: url = "https://seleniumbase.io/other/shadow_dom" sb.activate_cdp_mode(url) element = sb.cdp.get_nested_element("fancy-tabs", "section") print(element.text)
Once you have an element, you can use the CDP Element API to do more, such as element.click().
Related methods:
SeleniumBase/examples/cdp_mode/ReadMe.md
Lines 401 to 402 in 8317bc8
Beta Was this translation helpful? Give feedback.
All reactions
-
I used sb.cdp.get_nested_element(parent_selector, selector) to successfully to locate an element under an iframe.
It appears not to work with this particular shadow root configuration. Here are results from two of my test functions for troubleshooting:
self.selbase.cdp_find_nested_element("div[id="modal-div"]", "onboarding-modal[brand="TORI"]")
input("dev pause 1...")
self.selbase.cdp_find_nested_element("onboarding-modal[brand="TORI"]", "div[role="dialog"]")
input("dev pause 2...")
Console output:
(debug): get_nested_element(): searching (onboarding-modal[brand="TORI"]) in parent (div[id="modal-div"]) => located! => webdriver is in undetected mode
element= = <onboarding-modal silent="True" locale="fi" messages="{}" brand="TORI" clear_input="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1F560>" click="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1DB20>" flash="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1DBC0>" focus="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1DC60>" gui_click="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1DD00>" highlight_overlay="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1DDA0>" mouse_click="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1DE40>" mouse_drag="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1DEE0>" mouse_move="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1DF80>" press_keys="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1E020>" query_selector="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1E0C0>" querySelector="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1E0C0>" query_selector_all="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1E160>" querySelectorAll="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1E160>" remove_from_dom="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1E200>" save_screenshot="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1E2A0>" save_to_dom="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1FD80>" scroll_into_view="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1FE20>" select_option="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1FEC0>" send_file="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1FF60>" send_keys="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6D780040>" set_text="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6D7800E0>" set_value="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6D780180>" type="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6D780220>" get_position="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6D7802C0>" get_html="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6D780360>" get_js_attributes="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6D780400>" get_attribute="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6D7804A0>" get_parent="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6D780540>"></onboarding-modal>
dev pause 1...
(debug): get_nested_element(): searching (div[role="dialog"]) in parent (onboarding-modal[brand="TORI"]) => located! => webdriver is in undetected mode
element= = None
dev pause 2...
Beta Was this translation helpful? Give feedback.
All reactions
-
I tried the solution using the following code:
from seleniumbase import Driver
link = "https://www.yell.com/ucs/UcsSearchAction.do?keywords=accountants&location=newport-gwent&pageNum=1"
driver = Driver(uc=True, headless=False)
driver.uc_activate_cdp_mode(link)
driver.cdp.get_nested_element("div#usercentrics-root", "button[data-testid='uc-accept-all-button']").click()
driver.quit()
But i still get an error - the element in the shadow root is not found:
(seleniumBase) C:\DEVNEU\Fiverr2025\PROGRAMS\Yell_UK>python test.py
Traceb'ack (most recent call last):
File "C:\DEVNEU\Fiverr2025\PROGRAMS\Yell_UK\test.py", line 7, in <module>
driver.cdp.get_nested_element("div#usercentrics-root", "button[data-testid='uc-accept-all-button']").click()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'click'
Beta Was this translation helpful? Give feedback.
All reactions
-
sb.cdp.gui_press_keys("\t\t\t\t ") will press that button.
Beta Was this translation helpful? Give feedback.
All reactions
-
I tried this with the following code:
from seleniumbase import Driver
link = "https://www.yell.com/ucs/UcsSearchAction.do?keywords=accountants&location=newport-gwent&pageNum=1"
driver = Driver(uc=True, headless=False)
driver.uc_activate_cdp_mode(link)
driver.cdp.sleep(1)
driver.cdp.gui_press_keys("\t\t\t\t ")
input("Press1")
driver.quit()
But its still not working - i get this screen after the program run
Beta Was this translation helpful? Give feedback.
All reactions
-
from seleniumbase import SB with SB(uc=True, test=True, ad_block=True) as sb: url = "https://www.yell.com/ucs/UcsSearchAction.do?keywords=accountants&location=newport-gwent&pageNum=1" sb.activate_cdp_mode(url) sb.sleep(2) sb.uc_gui_click_captcha() sb.sleep(2) for i in range(8): sb.uc_gui_press_keys("\t") sb.sleep(0.5) sb.uc_gui_press_keys(" ") breakpoint()
Adjust timing as needed.
Websites may also change based on GeoLocation, so update as needed there.
Beta Was this translation helpful? Give feedback.
All reactions
-
Hello - it seems get_nested_element is still not working - find attached this example -
from seleniumbase import Driver
link = "https://www.daynurseries.co.uk/daynursery.cfm/searchazref/65432200502"
driver = Driver(uc=True, headless=False)
driver.uc_activate_cdp_mode(link)
input("Press!")
driver.cdp.get_nested_element("div[class='needsclick']", "button[aria-label='Accept All']").click()
input("Press")
I get this error:
(seleniumbase) C:\DEV\Fiverr2025\TRY\carystt>python test.py
Press!
Traceback (most recent call last):
File "C:\DEV\Fiverr2025\TRY\carystt\test.py", line 9, in <module>
driver.cdp.get_nested_element("div[class='needsclick']", "button[aria-label='Accept All']").click()
AttributeError: 'NoneType' object has no attribute 'click'
Is the function "get_nested_element" still not working or am i doing something wrong?
Beta Was this translation helpful? Give feedback.
All reactions
-
Some open shadow-roots can't be penetrated via CDP. (Closed shadow-root can never be penetrated.)
You may have to use PyAutoGUI as a workaround. Maybe count the number of times you need to press TAB before clicking SPACE or ENTER to trigger the button. There are some examples of that, eg:
SeleniumBase/examples/raw_hobbit.py
Line 7 in 0694314
Beta Was this translation helpful? Give feedback.