1

Over the past few days, I’ve been developing a project in C# (.NET 4.8) that scrapes and validates hundreds of HTTP/S proxies.

I’ve experimented with multiple approaches to confirm whether a proxy is alive, but with my current implementation (shown below), I’m only seeing roughly a 50% success rate when I cross check my results against popular online proxy checker tools like https://hide.mn/en/proxy-checker/ and https://proxy-seller.com/tools/proxy-checker/

Currently, I’m using an HttpClient configured with each proxy to make a request to https://api.ipify.org?format=json, which returns the public IP address. If the returned IP matches the proxy, I consider the proxy alive. However, even with this approach, only about half of the proxies that my program marks as "alive" are confirmed by the online proxy checker tools I listed above and no I'm not being rate limited by api.ipify.org!

I’m sharing my current method below and would appreciate any advice or suggestions for improving the accuracy!!

private static async Task<bool> Check(string proxy)
{
 string[] proxyParts = proxy.Split(':');
 if (proxyParts.Length < 2)
 {
 return false; // Invalid proxy format
 }
 string proxyIp = proxyParts[0];
 for (int attempt = 1; attempt <= retries; attempt++)
 {
 var handler = new HttpClientHandler
 {
 Proxy = new WebProxy(proxy),
 UseProxy = true
 };
 using (var checkClient = new HttpClient(handler) { Timeout = TimeSpan.FromMilliseconds(timeOut) })
 {
 try
 {
 string jsonResponse = await checkClient.GetStringAsync("https://api.ipify.org?format=json").ConfigureAwait(false);
 var jsonObject = JToken.Parse(jsonResponse);
 string publicIp = jsonObject["ip"]?.ToString();
 if (!string.IsNullOrEmpty(publicIp) && publicIp == proxyIp)
 {
 return true; // Proxy IP matches public IP
 }
 }
 catch
 {
 if (attempt == retries)
 {
 return false; // All retries failed
 }
 }
 await Task.Delay(delayMs).ConfigureAwait(false);
 }
 }
 return false;
}

I’ve tried a couple of ways to check the proxies. First, I make a request through each proxy to https://api.ipify.org?format=json and see if the returned IP matches the proxy, which should indicate it’s working. I also tried just sending a request to a regular URL and checking for a 200 response, thinking that might be more reliable. In both cases, I expected most of the working proxies to be flagged correctly, but the success rate is still only about half compared to online proxy checkers.

asked Aug 30 at 0:45
3
  • Suggestion: look at what is in the exception: is this a timeout or some HTTP error response. In the latter case maybe the proxy is rejecting your request (eg. it needs authentication). Commented Aug 30 at 7:10
  • Besides the exception check, you might want to also logging the case that the publicIp is different from proxyIp to determine if there are lots of such cases. After all, IP address is something that can be manipulated easily. Commented Aug 30 at 22:51
  • Just as an aside, you should not be creating and disposing multiple Http clients (if you don't already know why, search for HttpClient and sockets). Even in .Net Framework you can set up a singleton pattern. Commented Sep 1 at 16:46

0

Know someone who can answer? Share a link to this question via email, Twitter, or Facebook.

Your Answer

Draft saved
Draft discarded

Sign up or log in

Sign up using Google
Sign up using Email and Password

Post as a guest

Required, but never shown

Post as a guest

Required, but never shown

By clicking "Post Your Answer", you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.