Over the past few days, I’ve been developing a project in C# (.NET 4.8) that scrapes and validates hundreds of HTTP/S proxies.
I’ve experimented with multiple approaches to confirm whether a proxy is alive, but with my current implementation (shown below), I’m only seeing roughly a 50% success rate when I cross check my results against popular online proxy checker tools like https://hide.mn/en/proxy-checker/ and https://proxy-seller.com/tools/proxy-checker/
Currently, I’m using an HttpClient configured with each proxy to make a request to https://api.ipify.org?format=json, which returns the public IP address. If the returned IP matches the proxy, I consider the proxy alive. However, even with this approach, only about half of the proxies that my program marks as "alive" are confirmed by the online proxy checker tools I listed above and no I'm not being rate limited by api.ipify.org!
I’m sharing my current method below and would appreciate any advice or suggestions for improving the accuracy!!
private static async Task<bool> Check(string proxy)
{
string[] proxyParts = proxy.Split(':');
if (proxyParts.Length < 2)
{
return false; // Invalid proxy format
}
string proxyIp = proxyParts[0];
for (int attempt = 1; attempt <= retries; attempt++)
{
var handler = new HttpClientHandler
{
Proxy = new WebProxy(proxy),
UseProxy = true
};
using (var checkClient = new HttpClient(handler) { Timeout = TimeSpan.FromMilliseconds(timeOut) })
{
try
{
string jsonResponse = await checkClient.GetStringAsync("https://api.ipify.org?format=json").ConfigureAwait(false);
var jsonObject = JToken.Parse(jsonResponse);
string publicIp = jsonObject["ip"]?.ToString();
if (!string.IsNullOrEmpty(publicIp) && publicIp == proxyIp)
{
return true; // Proxy IP matches public IP
}
}
catch
{
if (attempt == retries)
{
return false; // All retries failed
}
}
await Task.Delay(delayMs).ConfigureAwait(false);
}
}
return false;
}
I’ve tried a couple of ways to check the proxies. First, I make a request through each proxy to https://api.ipify.org?format=json and see if the returned IP matches the proxy, which should indicate it’s working. I also tried just sending a request to a regular URL and checking for a 200 response, thinking that might be more reliable. In both cases, I expected most of the working proxies to be flagged correctly, but the success rate is still only about half compared to online proxy checkers.
-
Suggestion: look at what is in the exception: is this a timeout or some HTTP error response. In the latter case maybe the proxy is rejecting your request (eg. it needs authentication).Richard– Richard2025年08月30日 07:10:48 +00:00Commented Aug 30 at 7:10
-
Besides the exception check, you might want to also logging the case that the publicIp is different from proxyIp to determine if there are lots of such cases. After all, IP address is something that can be manipulated easily.victor6510– victor65102025年08月30日 22:51:08 +00:00Commented Aug 30 at 22:51
-
Just as an aside, you should not be creating and disposing multiple Http clients (if you don't already know why, search for HttpClient and sockets). Even in .Net Framework you can set up a singleton pattern.Jonathan Willcock– Jonathan Willcock2025年09月01日 16:46:03 +00:00Commented Sep 1 at 16:46