Python Requests doesnt work for https proxy

Question

I try to use https proxy in python like this:

proxiesDict ={
 'http': 'http://' + proxy_line,
 'https': 'https://' + proxy_line
}
response = requests.get('https://api.ipify.org/?format=json', proxies=proxiesDict, allow_redirects=False)

proxy_line is a proxy read from file in the format of ip:port. I checked this https proxy in browser and it works. But in python this code hangs for a few seconds and then i get exception:

HTTPSConnectionPool(host='api.ipify.org', port=443): Max retries exceeded with url: /?format=json (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x0425E450>: Failed to establish a new connection: [WinError 10060] 

I tried to use socks5 proxy, and it works on socks5 proxies with a PySocks installed. But for https i get this exception, can someone help me

• cs95: Remove the http:// and https://? Also, what does a sample proxy_line contain?
• Mrs_HillaryClinTon: @COLDSPEED Thanks! I think it works now, but it gives me not the proxy ip for some, but other ip. Its not my ip, but its not a proxy ip either, in the browser with proxy i visit same site and see the proxy ip. proxy_line contains a proxy like this: 79.110.31.87:8085, and in the browser i see the same ip when visiting api.ipify.org, but from python i get weird addresses
• cs95: It might have something to do with allow_redirects=False. In your browser, redirects are allowed automatically. Let me know if removing that fixes it, I'll create an answer for posterity.
• Mrs_HillaryClinTon: @cᴏʟᴅsᴘᴇᴇᴅ i tried to set allow_redirects=True and it still same weird results, for example for proxy 79.110.31.87:8085 i got {"ip":"162.220.246.230"} i tried to ,make request to whoer.net and it display the same ip
• Mrs_HillaryClinTon: @cᴏʟᴅsᴘᴇᴇᴅ nevermind, i had a system proxy set up. I removed it and now i always get my ip address instead of proxy from python

Answer 1

When specifying a proxy list for requests, the key is the protocol, and the value is the domain/ip. You don't need to specify http:// or https:// again, for the actual value.

So, your proxiesDict will be:

proxiesDict = {
 'http': proxy_line,
 'https': proxy_line
}

• aJetHorn: I am having the same issue as above except with a username:password proxy. Your answer is different from the Requests docs, which specifies to prepend 'http://'.
• user14603676: Requests Err:「Proxy URL had no scheme, should start with http:// or https://」
• Diego Suarez: That error is because the urllib3 version installed in your system, the one I had, did the same thing until I downgraded to urllib3==1.25.8, the newest one requires http scheme, the old one doesn't , and some said it has bugs.

Answer 2

You can also configure proxies by setting the enviroment variables:

$ export HTTP_PROXY="http://proxyIP:PORT"
$ export HTTPS_PROXY="http://proxyIP:PORT"

Then, you only need to execute your python script without proxy request.

Also, you can configure your proxy with http://user:password@host

For more information see this documentation: http://docs.python-requests.org/en/master/user/advanced/

• Mrs_HillaryClinTon: hello i need to read proxies from file, thats why i need to configure that way

Answer 3

Try using pycurl this function may help:

import pycurl
def pycurl_downloader(url, proxy_url, proxy_usr):
 """
 Download files with pycurl
 the proxy configuration:
 proxy_url = 'http://10.0.0.0:3128'
 proxy_usr = 'user:password'
 """
 c = pycurl.Curl()
 c.setopt(pycurl.FOLLOWLOCATION, 1)
 c.setopt(pycurl.MAXREDIRS, 5)
 c.setopt(pycurl.CONNECTTIMEOUT, 30)
 c.setopt(pycurl.AUTOREFERER, 1)
 if proxy_url: c.setopt(pycurl.PROXY, proxy_url)
 if proxy_usr: c.setopt(pycurl.PROXYUSERPWD, proxy_usr)
 content = StringIO()
 c.setopt(pycurl.URL, url)
 c.setopt(c.WRITEFUNCTION, content.write)
 try:
 c.perform()
 c.close()
 except pycurl.error, error:
 errno, errstr = error
 print 'An error occurred: ', errstr
 return content.getvalue()

• Mrs_HillaryClinTon: i'm gonna give it a try
• Austin L: I think there are typos in if proxy: that var does not exist. Also conten.write