Selenium: how to move physical mouse pointer outside the browser window

Question

I am using selenium (C#) with Grid to test web application. But I get unexpected results when the physical mouse is over the browser window.

Issue details
same question on stackoverflow.

I would like to move the physical mouse outside of browser window. I am not sure how to achieve it. I am using selenium grid to run tests on remote machines, so I would require something that would work over the Grid.

• Tim TY: Sikuli can be combined with Selenium and can do what you need. check for more details.

Answer 1

Selenium only works with a browser-driver. So you cannot move outside the working area, i.e. your browser.

You can use java method Actions to perform similar kind of tasks. If you have the coordinate where to click, you can try this.

Actions builder = new Actions(driver); 
builder.moveToElement(element, X, Y).click().build().perform();

Hope this helps.

• Michael Durrant: Won't element still need to be inside the browser window?
• H.D.: I have already tried this, it did not work. My issue is with the physical mouse pointer. Selenium will not work on that.

Answer 2

use java robot class or autoIT to move mouse. since you are using selenium grid you need to consider about screen resolution of multiple monitors.

both use screen coordinates to move mouse. you can use if commands inside the code to check the screen resolution and move mouse to relevant location. ex: if(resolutions is this) move to -> x1,y1 else (....) move to -> x2,y2

I hope this would work :)

• H.D.: I am still not sure how I can use it over selenium grid. How can I execute this piece of code remotely with grid?
• Suyama: add this in to ur test script so it will execute via selenium grid. Selenium grid distributes then execution into multiple machines, but it will execute your test script code. In the test script code call a method or a class to do this.
• H.D.: Its not going to work. Selenium Grid only works with webdriver commands. Any external script or code will not work over grid.

Answer 3

Java Robot class might help you, though it's also a good candidate to induce flakiness to your test suite. The class itself allows you to move the physical mouse around the screen (so using it on a server with no mouse attached might result in problems) and, of course, has no context to evaluate if the given coordinates are really matching the place you want it to be. Using it via the grid might be problematic, tbh, I have not yet had much experience with distributed environments but if you want to stay in Java I think it might be your only solution since Selenium lives inside your Browsers sandbox and can't (and honestly, shouldn't) act outside of it.

Here is the documentation for javas robot, though like I said, try to keep it as a last solution.