Python requests_html submit a form by clicking a button using JQuery

Question

I'm exploring Kenneth Reitz's requests_html and trying to submit a form of a JS Rendered Webpage using Jquery. I'm not sure how to do it but, here is my attempt:

from requests_html import HTMLSession
url = "https://example.com"
session = HTMLSession()
r = session.get(url)
r.html.render()
script = """
 () => {
 $("#some_input_field").val("Some value");
 $("#submit_button").click();
 }
 """
r.html.render(script=script, reload=False)

But, the value is not getting set on the input field & it isn't submitting the form... Is there any way to simulate button click or, form submit via xhr in requests_html?

For example: If we use selenium we can simulate button click pretty easily by typing:

element.click()

Answer 1

Ok, The following code is working in my case:

from requests_html import HTMLSession
url = "https://example.com"
session = HTMLSession()
r = session.get(url)
r.html.render()
script = """
 () => {
 if ( jQuery.isReady ) { 
 $("#some_input_field").val("Some value");
 $("#submit_button").click();
 }
 }
 """
r.html.render(script=script, reload=False)

EDIT: A better approach should be:

from requests_html import HTMLSession
url = "https://example.com"
session = HTMLSession()
r = session.get(url)
r.html.render()
script = """
 () => {
 $(document).ready(function() { 
 $("#some_input_field").val("Some value");
 $("#submit_button").click();
 })
 }
 """
r.html.render(script=script, reload=False)

• Jia Gao: Hey very much appreciate seeing this, could you please explain why and how the part after script= works?
• Wasi: @JasonGoal script variable is a string containing JavaScript code. It fills an input field and clicks the submit button. ()=>{} is an arrow function. The dollar sign in the second function is JQuery
• Jake: @Wasi Can you explain what document refers to in the second code snippit? I get the following error when I try to run (and have replaced the some_input_field and submit_button...: pyppeteer.errors.ElementHandleError: Evaluation failed: ReferenceError: $ is not defined...
• Jake: My script is as follows: ``` script = """ () => { $(document).ready(function() { $("#publish-to-mobile-button").click(); $("#publish-to-mobile-confirm-button").click(); }) } """ ```
• Jake: UPDATE: I figured it out with the help of github.com/psf/requests-html/issues/355#issuecomment-942914999

Answer 2

The below statements were not made by me but were helpful in resolving. Credit to alairjunior's response on this issue

I wasn't able to use the 'load' and 'DOMContentLoaded' events. Did not investigate why yet. I suspect the script is run using the "console", so it cannot get those events. But this is just a wild guess.

For a quick and dirty solution, I was able to use setTimeout:

script = """
() => {
setTimeout(function(){
 document.querySelectorAll("a")[2].click();
}, 3000);
}
"""

If I use r.html.render(sleep=10,script=script) I am able to get the content of the page after the click was executed.

Hope this is useful.