So this is in regards to scraping yes; no language in particular. Some sites allow you to see a JSON modal if you pull it directly from a web browser. But, at any notion a program is used, immediately declines the request and asks for an API? How does the site know the difference between user request and say a request from a CLI or selenium?
-
Some related posts: JavaScript only validation on AJAX form submit and How do hackers trick frontend validation?. In particular, note the various methods mentioned in the answers for creating requests that look identical to the intended front-end's requests.8bittree– 8bittree2021年02月05日 20:42:13 +00:00Commented Feb 5, 2021 at 20:42
1 Answer 1
In general, websites look for anomalous traffic patterns. Cookies that don't match up, requests made in the wrong order, that sort of thing.
If a website is keenly interested in this distinction, they can find out if you're a human user by presenting a challenge that is hard for automated programs to negotiate, like a Captcha.
There are other techniques they can employ to limit "bots," like rate limiting.
Beyond that, some websites look at things like the user agent string and session tokens, things that can be easily defeated by a savvy scraper. Generally, what you want to do is look at the Internet traffic using WireShark or Fiddler, and mimic the traffic that the web browser produces. Selenium doesn't do that out of the box.
-
So technically a simple scrape could take place if you had authenticated with the site, or would the rejected the request becomes it would not be common request made by that typical user? Edit: By mimic you mean creating a custom header data and cookie to communicate with the site?herboren– herboren2021年02月05日 19:12:08 +00:00Commented Feb 5, 2021 at 19:12
-
1
By mimic you mean creating a custom header data and cookie to communicate with the site?-- Yes, in the same way that the browser would.Robert Harvey– Robert Harvey2021年02月05日 19:14:45 +00:00Commented Feb 5, 2021 at 19:14